Max level shown:
12025-04-30T06:42:44.570ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_crdb
22025-04-30T06:42:44.571ZDEBGtest_snapshot_expunge: Starting CRDB
32025-04-30T06:42:44.607ZINFOtest_snapshot_expunge: cockroach temporary directory: /var/tmp/omicron_tmp/.tmpm60wLF
42025-04-30T06:42:44.607ZINFOtest_snapshot_expunge: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmpm60wLF/data)
52025-04-30T06:42:44.638ZINFOtest_snapshot_expunge: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmpm60wLF/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmpm60wLF/listen-url
62025-04-30T06:42:44.639ZINFOtest_snapshot_expunge: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=1 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:42:46.283ZINFOtest_snapshot_expunge: cockroach pid: 27039
82025-04-30T06:42:46.283ZINFOtest_snapshot_expunge: cockroach listen URL: postgresql://root@[::1]:34436/omicron?sslmode=disable
92025-04-30T06:42:46.283ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_clickhouse
102025-04-30T06:42:46.283ZDEBGtest_snapshot_expunge: Starting Clickhouse
112025-04-30T06:42:46.902ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:42:46.902ZDEBGtest_snapshot_expunge: Starting Management Gateway
132025-04-30T06:42:47.026ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:42:47.026ZDEBGtest_snapshot_expunge: Starting Dendrite for switch0
152025-04-30T06:42:47.073ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:42:47.073ZDEBGtest_snapshot_expunge: Starting mgd for switch0
172025-04-30T06:42:47.174ZDEBGtest_snapshot_expunge: mgd port is 36404
182025-04-30T06:42:47.174ZDEBGtest_snapshot_expunge (init_with_steps): Running step record_switch_dns
192025-04-30T06:42:47.175ZDEBGtest_snapshot_expunge: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:42:47.175ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_internal_dns
212025-04-30T06:42:47.175ZINFOtest_snapshot_expunge (store): opening sled database path = /var/tmp/omicron_tmp/.tmpoN8RnG
222025-04-30T06:42:47.299ZINFOtest_snapshot_expunge (store): pruning trees for generations newer than 0
232025-04-30T06:42:47.299ZINFOtest_snapshot_expunge (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:42:47.299ZINFOtest_snapshot_expunge (dns): DNS server bound to address kind = dns local_address = [::1]:42353
252025-04-30T06:42:47.300ZDEBGtest_snapshot_expunge (http): registered endpoint kind = dns local_addr = [::1]:58975 method = GET path = /config versions = all
262025-04-30T06:42:47.300ZDEBGtest_snapshot_expunge (http): registered endpoint kind = dns local_addr = [::1]:58975 method = PUT path = /config versions = all
272025-04-30T06:42:47.300ZINFOtest_snapshot_expunge (http): listening kind = dns local_addr = [::1]:58975
282025-04-30T06:42:47.302ZDEBGtest_snapshot_expunge (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:58975
292025-04-30T06:42:47.308ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_external_dns
302025-04-30T06:42:47.310ZINFOtest_snapshot_expunge (store): opening sled database path = /var/tmp/omicron_tmp/.tmpKi8Yqh
312025-04-30T06:42:47.328ZINFOtest_snapshot_expunge (store): pruning trees for generations newer than 0
322025-04-30T06:42:47.328ZINFOtest_snapshot_expunge (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:42:47.328ZINFOtest_snapshot_expunge (dns): DNS server bound to address kind = dns local_address = [::1]:35950
342025-04-30T06:42:47.328ZDEBGtest_snapshot_expunge (http): registered endpoint kind = dns local_addr = [::1]:46413 method = GET path = /config versions = all
352025-04-30T06:42:47.328ZDEBGtest_snapshot_expunge (http): registered endpoint kind = dns local_addr = [::1]:46413 method = PUT path = /config versions = all
362025-04-30T06:42:47.328ZINFOtest_snapshot_expunge (http): listening kind = dns local_addr = [::1]:46413
372025-04-30T06:42:47.329ZDEBGtest_snapshot_expunge (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:46413
382025-04-30T06:42:47.329ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_nexus_internal
392025-04-30T06:42:47.329ZDEBGtest_snapshot_expunge: Starting Nexus (internal API)
402025-04-30T06:42:47.330ZINFOtest_snapshot_expunge: setting up nexus server
412025-04-30T06:42:47.332ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Action
422025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Database
462025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Fleet
482025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Inventory
492025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Project
582025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Disk
592025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:42:47.333ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Instance
642025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = IpPool
652025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Vpc
672025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Image
752025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Certificate
802025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Rack
852025-04-30T06:42:47.334ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = SshKey
872025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Silo
882025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Sled
942025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = Service
1012025-04-30T06:42:47.335ZINFOtest_snapshot_expunge (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:42:47.336ZINFOtest_snapshot_expunge (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:42:47.363ZINFOtest_snapshot_expunge (ServerContext): Setting up resolver using DNS address: [::1]:42353
1042025-04-30T06:42:47.363ZINFOtest_snapshot_expunge (DnsResolver): new DNS resolver addresses = [[::1]:42353]
1052025-04-30T06:42:47.363ZINFOtest_snapshot_expunge (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:34436/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 34436,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:42:47.366ZDEBGtest_snapshot_expunge (ServerContext): registered USDT probes
1072025-04-30T06:42:47.406ZINFOtest_snapshot_expunge (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:42:47.496ZDEBGtest_snapshot_expunge (clickhouse-client): registered USDT probes id = 208fcc41-7c46-47c4-8a86-455b86525c8c
1092025-04-30T06:42:47.589ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /bgtasks versions = all
1102025-04-30T06:42:47.589ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:42:47.589ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /demo-saga versions = all
1202025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:42:47.590ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /metrics/producers versions = all
1442025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /mgs-updates versions = all
1452025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /sagas versions = all
1522025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /sleds/add versions = all
1572025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = GET path = /v1/ping versions = all
1612025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:42:47.591ZDEBGtest_snapshot_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:41836 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:42:47.591ZINFOtest_snapshot_expunge (dropshot_internal): listening local_addr = 127.0.0.1:41836
1642025-04-30T06:42:47.592ZDEBGtest_snapshot_expunge (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:41836
1652025-04-30T06:42:47.593ZINFOtest_snapshot_expunge (SEC): SEC running sec_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
1662025-04-30T06:42:47.599ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:42:47.599ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:42:47.599ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:42:47.600ZDEBGtest_snapshot_expunge (DataLoader): attempting to create built-in users
1702025-04-30T06:42:47.609ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:42:47.610ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:42:47.611ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:42:47.611ZINFOtest_snapshot_expunge (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:42:47.614ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 2e bb 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:60546 req_id = 94fc5290-91d1-4679-b19f-48c84c2ebee3
1752025-04-30T06:42:47.614ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11963,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60546 req_id = 94fc5290-91d1-4679-b19f-48c84c2ebee3
1762025-04-30T06:42:47.617ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11963, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60546 req_id = 94fc5290-91d1-4679-b19f-48c84c2ebee3
1772025-04-30T06:42:47.631ZINFOtest_snapshot_expunge (DataLoader): created 5 built-in users
1782025-04-30T06:42:47.632ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:42:47.632ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:42:47.635ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:42:47.635ZDEBGtest_snapshot_expunge (DataLoader): attempting to create built-in roles
1822025-04-30T06:42:47.635ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:42:47.635ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:42:47.635ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:42:47.645ZINFOtest_snapshot_expunge (DataLoader): created 10 built-in roles
1862025-04-30T06:42:47.645ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:42:47.645ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:42:47.647ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:42:47.647ZDEBGtest_snapshot_expunge (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:42:47.647ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:42:47.647ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:42:47.648ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:42:47.992ZINFOtest_snapshot_expunge (DataLoader): created 4 built-in role assignments
1942025-04-30T06:42:47.992ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:42:47.992ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:42:47.995ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:42:47.995ZDEBGtest_snapshot_expunge (DataLoader): attempting to create built-in silos
1982025-04-30T06:42:47.995ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:42:47.995ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:42:47.995ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:42:48.219ZINFOtest_snapshot_expunge (DataLoader): created 2 built-in silos
2022025-04-30T06:42:48.220ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:42:48.220ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:42:48.221ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:42:48.255ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:42:48.255ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:42:48.258ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:42:48.258ZDEBGtest_snapshot_expunge (DataLoader): attempting to create built-in projects
2092025-04-30T06:42:48.259ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:42:48.259ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:42:48.259ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:42:48.267ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:42:48.267ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:42:48.268ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:42:48.268ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:42:48.269ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:42:48.364ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:42:48.364ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:42:48.364ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:42:48.365ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:42:48.369ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:42:48.373ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:42:48.373ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:42:48.373ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:42:48.373ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:42:48.373ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:42:48.374ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:42:48.377ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:42:48.377ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:42:48.377ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:42:48.377ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:42:48.380ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:42:48.383ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:42:48.384ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:42:48.384ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:42:48.384ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:42:48.408ZINFOtest_snapshot_expunge (DataLoader): created built-in services project
2382025-04-30T06:42:48.408ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:42:48.408ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:42:48.411ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:42:48.411ZDEBGtest_snapshot_expunge (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:42:48.412ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:42:48.412ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:42:48.412ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:42:48.417ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:42:48.417ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:42:48.417ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:42:48.421ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:42:48.422ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:42:48.422ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:42:48.422ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:42:48.422ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:42:48.425ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:42:48.425ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:42:48.425ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:42:48.425ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:42:48.428ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:42:48.428ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:42:48.428ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:42:48.429ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:42:48.431ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:42:48.442ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:42:48.442ZTRACtest_snapshot_expunge (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:42:48.443ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:42:48.443ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:42:48.443ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:42:48.443ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:42:48.443ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:42:48.448ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:42:48.448ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:42:48.448ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:42:48.449ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:42:48.459ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:42:48.461ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:42:48.461ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:42:48.461ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:42:48.465ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:42:48.478ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:42:48.478ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:42:48.479ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:42:48.479ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:42:48.502ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:42:48.503ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:42:48.503ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:42:48.503ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:42:48.504ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:42:48.513ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:42:48.513ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:42:48.513ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:42:48.516ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:42:48.526ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:42:48.526ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:42:48.526ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:42:48.527ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:42:48.530ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:42:48.539ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:42:48.544ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:42:48.544ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:42:48.546ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:42:48.557ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:42:48.558ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:42:48.565ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:42:48.577ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:42:48.577ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:42:48.578ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:42:48.584ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:42:48.584ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:42:48.584ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:42:48.585ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:42:48.585ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:42:48.596ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:42:48.597ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:42:48.597ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:42:48.598ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:42:48.608ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:42:48.608ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:42:48.608ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:42:48.610ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:42:48.636ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:42:48.650ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:42:48.650ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:42:48.650ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:42:48.651ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:42:48.712ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:42:48.712ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:42:48.712ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:42:48.712ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:42:48.713ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:42:48.717ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:42:48.717ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:42:48.717ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:42:48.717ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:42:48.720ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:42:48.720ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:42:48.720ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:42:48.720ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:42:48.724ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:42:48.739ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:42:48.739ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:42:48.739ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:42:48.740ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:42:48.808ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:42:48.808ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:42:48.808ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:42:48.808ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:42:48.810ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:42:48.818ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:42:48.818ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:42:48.818ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:42:48.819ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:42:48.821ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:42:48.821ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:42:48.821ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:42:48.824ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:42:48.832ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:42:48.863ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:42:48.863ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:42:48.863ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:42:48.864ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:42:48.883ZDEBGtest_snapshot_expunge (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:42:48.883ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:42:48.883ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:42:48.884ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:42:48.891ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:42:48.891ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:42:48.892ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:42:48.898ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:42:48.898ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:42:48.902ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:42:48.909ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:42:48.919ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:42:48.919ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:42:48.920ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:42:48.920ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:42:48.925ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:42:48.925ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:42:48.925ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:42:48.926ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:42:48.937ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:42:48.937ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:42:48.937ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:42:48.937ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:42:48.947ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:42:48.953ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:42:48.953ZTRACtest_snapshot_expunge (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:42:48.953ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:42:48.953ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:42:48.953ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:42:48.953ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:42:48.953ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:42:48.955ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:42:48.955ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:42:48.956ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:42:48.956ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:42:48.970ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:42:48.970ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:42:48.970ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:42:48.970ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:42:48.974ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:42:48.982ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:42:48.982ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:42:48.982ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:42:48.982ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:42:48.995ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:42:48.996ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:42:48.996ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:42:48.996ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:42:48.997ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:42:49.002ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:42:49.002ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:42:49.002ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:42:49.003ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:42:49.006ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:42:49.006ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:42:49.006ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:42:49.006ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:42:49.018ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:42:49.030ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:42:49.030ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:42:49.030ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:42:49.031ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:42:49.144ZINFOtest_snapshot_expunge (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:42:49.144ZDEBGtest_snapshot_expunge (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:42:49.144ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:42:49.144ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:42:49.145ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:42:49.152ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:42:49.152ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:42:49.153ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:42:49.175ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:42:49.175ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:42:49.177ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:42:49.179ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:42:49.179ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:42:49.179ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:42:49.179ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:42:49.182ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:42:49.185ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:42:49.185ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:42:49.185ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:42:49.186ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:42:49.193ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:42:49.193ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:42:49.193ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:42:49.195ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:42:49.202ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:42:49.211ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:42:49.211ZTRACtest_snapshot_expunge (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:42:49.211ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:42:49.211ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:42:49.212ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:42:49.226ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:42:49.226ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:42:49.226ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:42:49.226ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:42:49.226ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:42:49.230ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:42:49.230ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:42:49.230ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:42:49.231ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:42:49.237ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:42:49.237ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:42:49.237ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:42:49.238ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:42:49.240ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:42:49.250ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:42:49.258ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:42:49.258ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:42:49.258ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:42:49.258ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:42:49.259ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:42:49.262ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:42:49.262ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:42:49.262ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:42:49.270ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:42:49.286ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:42:49.286ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:42:49.287ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:42:49.288ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:42:49.292ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:42:49.304ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:42:49.305ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:42:49.305ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:42:49.305ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:42:49.317ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:42:49.319ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:42:49.320ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:42:49.335ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:42:49.335ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:42:49.335ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:42:49.335ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:42:49.336ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:42:49.348ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:42:49.348ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:42:49.349ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:42:49.350ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:42:49.353ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:42:49.353ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:42:49.353ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:42:49.362ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:42:49.365ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:42:49.374ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:42:49.380ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:42:49.380ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:42:49.380ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:42:49.380ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:42:49.381ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:42:49.385ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:42:49.385ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:42:49.385ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:42:49.386ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:42:49.391ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:42:49.391ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:42:49.391ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:42:49.392ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:42:49.394ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:42:49.405ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:42:49.405ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:42:49.405ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:42:49.405ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:42:49.483ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:42:49.483ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:42:49.485ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:42:49.489ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:42:49.489ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:42:49.489ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:42:49.489ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:42:49.490ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:42:49.501ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:42:49.501ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:42:49.501ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:42:49.501ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:42:49.511ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:42:49.511ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:42:49.511ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:42:49.513ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:42:49.517ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:42:49.525ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:42:49.542ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:42:49.542ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:42:49.542ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:42:49.542ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:42:49.543ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:42:49.551ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:42:49.551ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:42:49.551ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:42:49.551ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:42:49.551ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:42:49.551ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:42:49.551ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:42:49.551ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:42:49.553ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:42:49.562ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:42:49.562ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:42:49.562ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:42:49.563ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:42:49.569ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:42:49.569ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:42:49.571ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:42:49.575ZINFOtest_snapshot_expunge (DataLoader): created built-in services vpc subnets
5612025-04-30T06:42:49.575ZINFOtest_snapshot_expunge (DataLoader): created built-in services vpc
5622025-04-30T06:42:49.575ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:42:49.575ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:42:49.579ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:42:49.579ZDEBGtest_snapshot_expunge (DataLoader): attempting to create silo users
5662025-04-30T06:42:49.579ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:42:49.579ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:42:49.580ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:42:49.587ZINFOtest_snapshot_expunge (DataLoader): created 2 silo users
5702025-04-30T06:42:49.587ZDEBGtest_snapshot_expunge (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:42:49.587ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:42:49.587ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:42:49.588ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:42:49.592ZINFOtest_snapshot_expunge (DataLoader): created 2 silo user role assignments
5752025-04-30T06:42:49.592ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:42:49.592ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:42:49.593ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:42:49.598ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:42:49.598ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:42:49.599ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:42:49.610ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:42:49.610ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:42:49.610ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:42:49.610ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:42:49.610ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:42:49.614ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:42:49.617ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:42:49.617ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:42:49.617ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:42:49.618ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:42:49.625ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:42:49.625ZTRACtest_snapshot_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:42:49.625ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:42:49.625ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:42:49.626ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:42:49.633ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:42:49.648ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:42:49.648ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:42:49.648ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:42:49.649ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:42:49.739ZTRACtest_snapshot_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:42:49.741ZDEBGtest_snapshot_expunge (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:42:49.741ZDEBGtest_snapshot_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:42:49.749ZINFOtest_snapshot_expunge (nexus): populate complete
6052025-04-30T06:42:49.749ZINFOtest_snapshot_expunge (nexus): activating background tasks
6062025-04-30T06:42:49.755ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_sled1
6072025-04-30T06:42:49.755ZINFOtest_snapshot_expunge (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:42:49.810ZINFOtest_snapshot_expunge (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:42:49.811ZINFOtest_snapshot_expunge (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:42:49.811ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:42:49.811ZDEBGtest_snapshot_expunge (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:42:49.811ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:42:49.811ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:42:49.817ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:42:49.820ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6162025-04-30T06:42:49.820ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6172025-04-30T06:42:49.821ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6182025-04-30T06:42:49.823ZWARNtest_snapshot_expunge (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6192025-04-30T06:42:49.823ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 1.634487ms iteration = 1 status = {"error":"no config nor servers"}
6202025-04-30T06:42:49.823ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6212025-04-30T06:42:49.823ZDEBGtest_snapshot_expunge (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6222025-04-30T06:42:49.823ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
6232025-04-30T06:42:49.823ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6242025-04-30T06:42:49.824ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6252025-04-30T06:42:49.825ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6262025-04-30T06:42:49.825ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6272025-04-30T06:42:49.827ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
6282025-04-30T06:42:49.827ZWARNtest_snapshot_expunge (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
6292025-04-30T06:42:49.827ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 55.2\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6302025-04-30T06:42:49.827ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
6312025-04-30T06:42:49.827ZINFOtest_snapshot_expunge (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:32:49.827214375 UTC
6322025-04-30T06:42:49.827ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
6332025-04-30T06:42:49.827ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
6342025-04-30T06:42:49.829ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
6352025-04-30T06:42:49.829ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
6362025-04-30T06:42:49.829ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
6372025-04-30T06:42:49.830ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6382025-04-30T06:42:49.833ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
6392025-04-30T06:42:49.833ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6402025-04-30T06:42:49.834ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6412025-04-30T06:42:49.834ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
6422025-04-30T06:42:49.834ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
6432025-04-30T06:42:49.834ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6442025-04-30T06:42:49.835ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6452025-04-30T06:42:49.835ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
6462025-04-30T06:42:49.835ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
6472025-04-30T06:42:49.835ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6482025-04-30T06:42:49.837ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6492025-04-30T06:42:49.837ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
6502025-04-30T06:42:49.837ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
6512025-04-30T06:42:49.837ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
6522025-04-30T06:42:49.837ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6532025-04-30T06:42:49.837ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
6542025-04-30T06:42:49.837ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6552025-04-30T06:42:49.838ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6562025-04-30T06:42:49.838ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
6572025-04-30T06:42:49.838ZWARNtest_snapshot_expunge (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
6582025-04-30T06:42:49.838ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 182.323\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6592025-04-30T06:42:49.838ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
6602025-04-30T06:42:49.838ZWARNtest_snapshot_expunge (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
6612025-04-30T06:42:49.838ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 35.089\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6622025-04-30T06:42:49.838ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
6632025-04-30T06:42:49.838ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
6642025-04-30T06:42:49.838ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6652025-04-30T06:42:49.838ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
6662025-04-30T06:42:49.838ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
6672025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
6682025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
6692025-04-30T06:42:49.839ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
6702025-04-30T06:42:49.839ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6712025-04-30T06:42:49.839ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
6722025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6732025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6742025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
6752025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 21.423\u{b5}s iteration = 1 status = {"error":"task disabled"}
6762025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
6772025-04-30T06:42:49.839ZWARNtest_snapshot_expunge (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
6782025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 15.831\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6792025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
6802025-04-30T06:42:49.839ZINFOtest_snapshot_expunge (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
6812025-04-30T06:42:49.839ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 79.939\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6822025-04-30T06:42:49.840ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
6832025-04-30T06:42:49.840ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
6842025-04-30T06:42:49.840ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6852025-04-30T06:42:49.840ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
6862025-04-30T06:42:49.840ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6872025-04-30T06:42:49.841ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6882025-04-30T06:42:49.841ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
6892025-04-30T06:42:49.841ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
6902025-04-30T06:42:49.841ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6912025-04-30T06:42:49.841ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
6922025-04-30T06:42:49.842ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6932025-04-30T06:42:49.842ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6942025-04-30T06:42:49.843ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
6952025-04-30T06:42:49.843ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
6962025-04-30T06:42:49.843ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6972025-04-30T06:42:49.844ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6982025-04-30T06:42:49.844ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
6992025-04-30T06:42:49.844ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7002025-04-30T06:42:49.844ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7012025-04-30T06:42:49.844ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7022025-04-30T06:42:49.844ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7032025-04-30T06:42:49.844ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7042025-04-30T06:42:49.844ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7052025-04-30T06:42:49.844ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7062025-04-30T06:42:49.845ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7072025-04-30T06:42:49.845ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7082025-04-30T06:42:49.845ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7092025-04-30T06:42:49.845ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7102025-04-30T06:42:49.845ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7112025-04-30T06:42:49.846ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7122025-04-30T06:42:49.846ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7132025-04-30T06:42:49.846ZINFOtest_snapshot_expunge (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7142025-04-30T06:42:49.846ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = instance_updater elapsed = 283.935\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7152025-04-30T06:42:49.846ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7162025-04-30T06:42:49.846ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7172025-04-30T06:42:49.846ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7182025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7192025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7202025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7212025-04-30T06:42:49.848ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
7222025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7232025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7242025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
7252025-04-30T06:42:49.848ZINFOtest_snapshot_expunge (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
7262025-04-30T06:42:49.848ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
7272025-04-30T06:42:49.848ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7282025-04-30T06:42:49.848ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
7292025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7302025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7312025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
7322025-04-30T06:42:49.848ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
7332025-04-30T06:42:49.848ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
7342025-04-30T06:42:49.849ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
7352025-04-30T06:42:49.849ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
7362025-04-30T06:42:49.850ZDEBGtest_snapshot_expunge (SagaRecoverer): listing candidate sagas for recovery
7372025-04-30T06:42:49.850ZTRACtest_snapshot_expunge (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
7382025-04-30T06:42:49.850ZDEBGtest_snapshot_expunge (SagaRecoverer): roles roles = RoleSet { roles: {} }
7392025-04-30T06:42:49.851ZDEBGtest_snapshot_expunge (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
7402025-04-30T06:42:49.851ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
7412025-04-30T06:42:49.851ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
7422025-04-30T06:42:49.851ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
7432025-04-30T06:42:49.851ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
7442025-04-30T06:42:49.851ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
7452025-04-30T06:42:49.851ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
7462025-04-30T06:42:49.851ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7472025-04-30T06:42:49.852ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7482025-04-30T06:42:49.852ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
7492025-04-30T06:42:49.852ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
7502025-04-30T06:42:49.852ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7512025-04-30T06:42:49.853ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
7522025-04-30T06:42:49.853ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
7532025-04-30T06:42:49.853ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
7542025-04-30T06:42:49.853ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7552025-04-30T06:42:49.853ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7562025-04-30T06:42:49.853ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
7572025-04-30T06:42:49.853ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
7582025-04-30T06:42:49.853ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7592025-04-30T06:42:49.854ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
7602025-04-30T06:42:49.854ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
7612025-04-30T06:42:49.854ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
7622025-04-30T06:42:49.854ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7632025-04-30T06:42:49.854ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
7642025-04-30T06:42:49.854ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7652025-04-30T06:42:49.855ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7662025-04-30T06:42:49.855ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
7672025-04-30T06:42:49.855ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
7682025-04-30T06:42:49.855ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7692025-04-30T06:42:49.855ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
7702025-04-30T06:42:49.855ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
7712025-04-30T06:42:49.855ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
7722025-04-30T06:42:49.855ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
7732025-04-30T06:42:49.856ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
7742025-04-30T06:42:49.856ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
7752025-04-30T06:42:49.856ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
7762025-04-30T06:42:49.856ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
7772025-04-30T06:42:49.857ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
7782025-04-30T06:42:49.858ZINFOtest_snapshot_expunge (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpTUreDe/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7792025-04-30T06:42:49.858ZDEBGtest_snapshot_expunge (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpTUreDe/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7802025-04-30T06:42:49.858ZINFOtest_snapshot_expunge (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpphPCl1/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7812025-04-30T06:42:49.858ZDEBGtest_snapshot_expunge (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpphPCl1/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7822025-04-30T06:42:49.949ZDEBGtest_snapshot_expunge (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:42164 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7832025-04-30T06:42:49.950ZINFOtest_snapshot_expunge (dropshot (Repo Depot)): listening local_addr = [::1]:42164 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7842025-04-30T06:42:49.951ZDEBGtest_snapshot_expunge (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:42164 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7852025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7862025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7872025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7882025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7892025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7902025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7912025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7922025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7932025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7942025-04-30T06:42:49.960ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7952025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7962025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7972025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7982025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7992025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8002025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8012025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8022025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8032025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8042025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8052025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8062025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8072025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8082025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8092025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8102025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8112025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8122025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8132025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8142025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8152025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8162025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8172025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8182025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8192025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8202025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8212025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8222025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8232025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8242025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8252025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8262025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8272025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8282025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8292025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8302025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8312025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8322025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8332025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8342025-04-30T06:42:49.961ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8352025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8362025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8372025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8382025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8392025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8402025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8412025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8422025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8432025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8442025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8452025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8462025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8472025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8482025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8492025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8502025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8512025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8522025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8532025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:51119 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8542025-04-30T06:42:49.962ZINFOtest_snapshot_expunge (dropshot): listening local_addr = [::1]:51119 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8552025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:51119 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8562025-04-30T06:42:49.962ZDEBGtest_snapshot_expunge (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8572025-04-30T06:42:49.963ZDEBGtest_snapshot_expunge (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:41836/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8582025-04-30T06:42:49.964ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 13 51 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 .Q..........._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:59288 req_id = 729e2a35-8282-4323-af5c-c37d6995420b
8592025-04-30T06:42:49.964ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4945,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59288 req_id = 729e2a35-8282-4323-af5c-c37d6995420b
8602025-04-30T06:42:49.965ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4945, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59288 req_id = 729e2a35-8282-4323-af5c-c37d6995420b
8612025-04-30T06:42:49.965ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 33 ee 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 3............_ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62985 req_id = 79d04bbd-8ebe-443f-a8a2-949dc0e4dc6e
8622025-04-30T06:42:49.965ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13294,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62985 req_id = 79d04bbd-8ebe-443f-a8a2-949dc0e4dc6e
8632025-04-30T06:42:49.965ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13294, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62985 req_id = 79d04bbd-8ebe-443f-a8a2-949dc0e4dc6e
8642025-04-30T06:42:49.965ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
8652025-04-30T06:42:49.965ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8662025-04-30T06:42:49.965ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
8672025-04-30T06:42:49.965ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8682025-04-30T06:42:49.966ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8692025-04-30T06:42:49.967ZINFOtest_snapshot_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:41836 remote_addr = 127.0.0.1:61932
8702025-04-30T06:42:49.967ZWARNtest_snapshot_expunge (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8712025-04-30T06:42:49.967ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 147.229235ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8722025-04-30T06:42:49.967ZWARNtest_snapshot_expunge (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8732025-04-30T06:42:49.968ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 142.949408ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8742025-04-30T06:42:49.968ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8752025-04-30T06:42:49.968ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8762025-04-30T06:42:49.968ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8772025-04-30T06:42:49.968ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8782025-04-30T06:42:49.969ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8792025-04-30T06:42:49.970ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:61932 req_id = a29ee37d-9059-4f82-983c-ee44c9a6e9d1 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8802025-04-30T06:42:49.972ZINFOtest_snapshot_expunge (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8812025-04-30T06:42:49.981ZINFOtest_snapshot_expunge (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:32:49.827214375 UTC pruned = {}
8822025-04-30T06:42:49.981ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 153.924545ms iteration = 1 status = {"expiration":"2025-04-30T06:32:49.827214375Z","pruned":[]}
8832025-04-30T06:42:49.985ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8842025-04-30T06:42:49.987ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 80 49 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .I..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56823 req_id = 67e9ef49-f3d2-4eaa-b75d-435544be65ff
8852025-04-30T06:42:49.988ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32841,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56823 req_id = 67e9ef49-f3d2-4eaa-b75d-435544be65ff
8862025-04-30T06:42:49.999ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32841, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56823 req_id = 67e9ef49-f3d2-4eaa-b75d-435544be65ff
8872025-04-30T06:42:50.019ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8882025-04-30T06:42:50.021ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 183.823447ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8892025-04-30T06:42:50.032ZERROtest_snapshot_expunge (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8902025-04-30T06:42:50.032ZERROtest_snapshot_expunge (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8912025-04-30T06:42:50.032ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 198.025503ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8922025-04-30T06:42:50.048ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8932025-04-30T06:42:50.093ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
8942025-04-30T06:42:50.093ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
8952025-04-30T06:42:50.094ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8962025-04-30T06:42:50.094ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8972025-04-30T06:42:50.102ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8982025-04-30T06:42:50.113ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8992025-04-30T06:42:50.113ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
9002025-04-30T06:42:50.113ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9012025-04-30T06:42:50.114ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9022025-04-30T06:42:50.114ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9032025-04-30T06:42:50.123ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9042025-04-30T06:42:50.158ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9052025-04-30T06:42:50.158ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9062025-04-30T06:42:50.159ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9072025-04-30T06:42:50.163ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9082025-04-30T06:42:50.195ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9092025-04-30T06:42:50.195ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9102025-04-30T06:42:50.195ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9112025-04-30T06:42:50.195ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9122025-04-30T06:42:50.196ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9132025-04-30T06:42:50.205ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9142025-04-30T06:42:50.205ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9152025-04-30T06:42:50.205ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9162025-04-30T06:42:50.205ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9172025-04-30T06:42:50.212ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 52 ca 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 R............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49207 req_id = c9765227-57b2-43c4-98a4-777cc6480aaf
9182025-04-30T06:42:50.212ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21194,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49207 req_id = c9765227-57b2-43c4-98a4-777cc6480aaf
9192025-04-30T06:42:50.213ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21194, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49207 req_id = c9765227-57b2-43c4-98a4-777cc6480aaf
9202025-04-30T06:42:50.214ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9212025-04-30T06:42:50.225ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9222025-04-30T06:42:50.225ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9232025-04-30T06:42:50.226ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9242025-04-30T06:42:50.231ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9252025-04-30T06:42:50.234ZERROtest_snapshot_expunge (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9262025-04-30T06:42:50.234ZERROtest_snapshot_expunge (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9272025-04-30T06:42:50.235ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 399.328897ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9282025-04-30T06:42:50.299ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9292025-04-30T06:42:50.299ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9302025-04-30T06:42:50.300ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9312025-04-30T06:42:50.334ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9322025-04-30T06:42:50.338ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9332025-04-30T06:42:50.338ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9342025-04-30T06:42:50.338ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9352025-04-30T06:42:50.338ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9362025-04-30T06:42:50.338ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9372025-04-30T06:42:50.339ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9382025-04-30T06:42:50.387ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 536.543739ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9392025-04-30T06:42:50.443ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9402025-04-30T06:42:50.443ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9412025-04-30T06:42:50.444ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9422025-04-30T06:42:50.447ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9432025-04-30T06:42:50.452ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9442025-04-30T06:42:50.452ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9452025-04-30T06:42:50.452ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9462025-04-30T06:42:50.452ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9472025-04-30T06:42:50.571ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9482025-04-30T06:42:50.571ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9492025-04-30T06:42:50.571ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9502025-04-30T06:42:50.574ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9512025-04-30T06:42:50.578ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9522025-04-30T06:42:50.578ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9532025-04-30T06:42:50.578ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9542025-04-30T06:42:50.578ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9552025-04-30T06:42:50.578ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9562025-04-30T06:42:50.578ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9572025-04-30T06:42:50.580ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 726.517259ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9582025-04-30T06:42:50.582ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
9592025-04-30T06:42:50.582ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9602025-04-30T06:42:50.582ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9612025-04-30T06:42:50.584ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 728.689216ms iteration = 1 status = {"by_rx":{},"error":null}
9622025-04-30T06:42:50.587ZINFOtest_snapshot_expunge (SagaRecoverer): listed in-progress sagas count = 0
9632025-04-30T06:42:50.587ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 738.367239ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9642025-04-30T06:42:50.588ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9652025-04-30T06:42:50.589ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9662025-04-30T06:42:50.589ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9672025-04-30T06:42:50.607ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9682025-04-30T06:42:50.620ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9692025-04-30T06:42:50.621ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9702025-04-30T06:42:50.621ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9712025-04-30T06:42:50.621ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9722025-04-30T06:42:50.643ZTRACtest_snapshot_expunge (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9732025-04-30T06:42:50.643ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9742025-04-30T06:42:50.643ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9752025-04-30T06:42:50.644ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9762025-04-30T06:42:50.662ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9772025-04-30T06:42:50.662ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9782025-04-30T06:42:50.662ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9792025-04-30T06:42:50.662ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9802025-04-30T06:42:50.663ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9812025-04-30T06:42:50.683ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 713426 local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:61932 req_id = a29ee37d-9059-4f82-983c-ee44c9a6e9d1 response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
9822025-04-30T06:42:50.685ZDEBGtest_snapshot_expunge (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:41836/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "a29ee37d-9059-4f82-983c-ee44c9a6e9d1", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9832025-04-30T06:42:50.685ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_crucible_pantry
9842025-04-30T06:42:50.686ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9852025-04-30T06:42:50.686ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9862025-04-30T06:42:50.686ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9872025-04-30T06:42:50.686ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9882025-04-30T06:42:50.686ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9892025-04-30T06:42:50.687ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9902025-04-30T06:42:50.687ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9912025-04-30T06:42:50.687ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9922025-04-30T06:42:50.687ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9932025-04-30T06:42:50.687ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9942025-04-30T06:42:50.687ZDEBGtest_snapshot_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:54156 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
9952025-04-30T06:42:50.687ZINFOtest_snapshot_expunge (dropshot): listening kind = pantry local_addr = [::1]:54156 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9962025-04-30T06:42:50.688ZDEBGtest_snapshot_expunge (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:54156 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9972025-04-30T06:42:50.688ZINFOtest_snapshot_expunge (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:54156 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9982025-04-30T06:42:50.688ZDEBGtest_snapshot_expunge (init_with_steps): Running step populate_internal_dns
9992025-04-30T06:42:50.688ZDEBGtest_snapshot_expunge: Populating Internal DNS
10002025-04-30T06:42:50.826ZINFOtest_snapshot_expunge: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:42:50.826045361Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 65174, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "58f2365f-b720-4ec1-ad50-a68fadcacb1c.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "69aed484-7768-456c-9a68-bca265ad4317.host": [ Aaaa( ::1, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "76bd4c4b-c66e-40e4-90b4-ef0afc6ed104.host.control-plane.oxide.internal", }, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "76bd4c4b-c66e-40e4-90b4-ef0afc6ed104.host.control-plane.oxide.internal", }, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 58975, target: "8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal", }, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 41836, target: "58f2365f-b720-4ec1-ad50-a68fadcacb1c.host.control-plane.oxide.internal", }, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 40004, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 54156, target: "fd382bc8-252a-498d-a152-387b7e5fb18a.host.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 46413, target: "69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 36404, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 38280, target: "76bd4c4b-c66e-40e4-90b4-ef0afc6ed104.host.control-plane.oxide.internal", }, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 34436, target: "76de6b0e-3a17-4e6b-9616-5814fc4102de.host.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "76bd4c4b-c66e-40e4-90b4-ef0afc6ed104.host.control-plane.oxide.internal", }, ), ], "76de6b0e-3a17-4e6b-9616-5814fc4102de.host": [ Aaaa( ::1, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "76bd4c4b-c66e-40e4-90b4-ef0afc6ed104.host": [ Aaaa( ::1, ), ], "8fa73ed5-840a-435c-8250-4547cafe1025.host": [ Aaaa( ::1, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "fd382bc8-252a-498d-a152-387b7e5fb18a.host": [ Aaaa( ::1, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 42164, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], }, }, ], }
10012025-04-30T06:42:50.826ZDEBGtest_snapshot_expunge: client request body = Some(Body) method = PUT uri = http://[::1]:58975/config
10022025-04-30T06:42:50.827ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
10032025-04-30T06:42:50.827ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 986.746629ms iteration = 1 status = {"error":"inventory collection is None"}
10042025-04-30T06:42:50.830ZDEBGtest_snapshot_expunge (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
10052025-04-30T06:42:50.830ZINFOtest_snapshot_expunge (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
10062025-04-30T06:42:50.830ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 981.930252ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
10072025-04-30T06:42:50.831ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10082025-04-30T06:42:50.842ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
10092025-04-30T06:42:50.843ZWARNtest_snapshot_expunge (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
10102025-04-30T06:42:50.843ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 1.005933587s iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
10112025-04-30T06:42:50.843ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
10122025-04-30T06:42:50.843ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
10132025-04-30T06:42:50.843ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 991.242165ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
10142025-04-30T06:42:50.844ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10152025-04-30T06:42:50.847ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10162025-04-30T06:42:50.847ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10172025-04-30T06:42:50.853ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 997.931323ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
10182025-04-30T06:42:50.853ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 1.011376933s iteration = 1 status = {}
10192025-04-30T06:42:50.853ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10202025-04-30T06:42:50.854ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10212025-04-30T06:42:50.854ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10222025-04-30T06:42:50.854ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10232025-04-30T06:42:50.855ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10242025-04-30T06:42:50.855ZWARNtest_snapshot_expunge (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
10252025-04-30T06:42:50.855ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 1.032661591s iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
10262025-04-30T06:42:50.856ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10272025-04-30T06:42:50.874ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10282025-04-30T06:42:50.874ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10292025-04-30T06:42:50.874ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10302025-04-30T06:42:50.875ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10312025-04-30T06:42:50.885ZINFOtest_snapshot_expunge (http): accepted connection kind = dns local_addr = [::1]:58975 remote_addr = [::1]:62799
10322025-04-30T06:42:50.889ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e6 61 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .a..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59506 req_id = 602ebe97-e6ae-4ed4-8d4e-969f25d0b5dd
10332025-04-30T06:42:50.890ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58977,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59506 req_id = 602ebe97-e6ae-4ed4-8d4e-969f25d0b5dd
10342025-04-30T06:42:50.890ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58977, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59506 req_id = 602ebe97-e6ae-4ed4-8d4e-969f25d0b5dd
10352025-04-30T06:42:50.913ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
10362025-04-30T06:42:50.913ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.075220637s iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
10372025-04-30T06:42:50.913ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10382025-04-30T06:42:50.913ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10392025-04-30T06:42:50.913ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10402025-04-30T06:42:50.913ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10412025-04-30T06:42:50.913ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10422025-04-30T06:42:50.915ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10432025-04-30T06:42:50.915ZTRACtest_snapshot_expunge (http): incoming request kind = dns local_addr = [::1]:58975 method = PUT remote_addr = [::1]:62799 req_id = ef63f1a0-f384-47a7-a02d-b9dfd76bf665 uri = /config
10442025-04-30T06:42:50.916ZINFOtest_snapshot_expunge (store): attempting generation update new_generation = 1 req_id = ef63f1a0-f384-47a7-a02d-b9dfd76bf665
10452025-04-30T06:42:50.916ZINFOtest_snapshot_expunge (store): pruning trees for generations newer than 0
10462025-04-30T06:42:50.916ZDEBGtest_snapshot_expunge (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10472025-04-30T06:42:50.917ZDEBGtest_snapshot_expunge (store): updating current config
10482025-04-30T06:42:50.917ZDEBGtest_snapshot_expunge (store): flushing default tree
10492025-04-30T06:42:50.917ZINFOtest_snapshot_expunge (store): pruning trees for generations older than 1 keep = 3
10502025-04-30T06:42:50.917ZINFOtest_snapshot_expunge (store): updated generation new_generation = 1 req_id = ef63f1a0-f384-47a7-a02d-b9dfd76bf665
10512025-04-30T06:42:50.918ZINFOtest_snapshot_expunge (http): request completed kind = dns latency_us = 2475 local_addr = [::1]:58975 method = PUT remote_addr = [::1]:62799 req_id = ef63f1a0-f384-47a7-a02d-b9dfd76bf665 response_code = 204 uri = /config
10522025-04-30T06:42:50.918ZDEBGtest_snapshot_expunge: client response result = Ok(Response { url: "http://[::1]:58975/config", status: 204, headers: {"x-request-id": "ef63f1a0-f384-47a7-a02d-b9dfd76bf665", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
10532025-04-30T06:42:50.919ZDEBGtest_snapshot_expunge (init_with_steps): Running step configure_sled_agents
10542025-04-30T06:42:51.022ZDEBGtest_snapshot_expunge: client request body = Some(Body) method = PUT uri = http://[::1]:51119/omicron-config
10552025-04-30T06:42:51.023ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10562025-04-30T06:42:51.028ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10572025-04-30T06:42:51.028ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10582025-04-30T06:42:51.028ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10592025-04-30T06:42:51.029ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10602025-04-30T06:42:51.029ZTRACtest_snapshot_expunge (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10612025-04-30T06:42:51.029ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10622025-04-30T06:42:51.029ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10632025-04-30T06:42:51.030ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10642025-04-30T06:42:51.030ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:40943 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10652025-04-30T06:42:51.036ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 1.191739373s iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10662025-04-30T06:42:51.037ZWARNtest_snapshot_expunge (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
10672025-04-30T06:42:51.037ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 1.225978793s iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
10682025-04-30T06:42:51.038ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = PUT remote_addr = [::1]:40943 req_id = 089709ae-8a7a-4af5-b671-282711e66836 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10692025-04-30T06:42:51.047ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 9522 local_addr = [::1]:51119 method = PUT remote_addr = [::1]:40943 req_id = 089709ae-8a7a-4af5-b671-282711e66836 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10702025-04-30T06:42:51.049ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = region_replacement elapsed = 1.205146246s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10712025-04-30T06:42:51.066ZINFOtest_snapshot_expunge (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10722025-04-30T06:42:51.066ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 1.22114402s iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10732025-04-30T06:42:51.072ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10742025-04-30T06:42:51.078ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10752025-04-30T06:42:51.078ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10762025-04-30T06:42:51.078ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10772025-04-30T06:42:51.079ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10782025-04-30T06:42:51.080ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10792025-04-30T06:42:51.080ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10802025-04-30T06:42:51.080ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10812025-04-30T06:42:51.080ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10822025-04-30T06:42:51.081ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10832025-04-30T06:42:51.082ZDEBGtest_snapshot_expunge: client response result = Ok(Response { url: "http://[::1]:51119/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "089709ae-8a7a-4af5-b671-282711e66836", "content-length": "26", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
10842025-04-30T06:42:51.082ZDEBGtest_snapshot_expunge: client request body = Some(Body) method = PUT uri = http://[::1]:51119/network-bootstore-config
10852025-04-30T06:42:51.086ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = PUT remote_addr = [::1]:40943 req_id = 492a549c-2a07-4825-8f6f-4b000af3d72b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10862025-04-30T06:42:51.087ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 944 local_addr = [::1]:51119 method = PUT remote_addr = [::1]:40943 req_id = 492a549c-2a07-4825-8f6f-4b000af3d72b response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10872025-04-30T06:42:51.099ZDEBGtest_snapshot_expunge: client response result = Ok(Response { url: "http://[::1]:51119/network-bootstore-config", status: 204, headers: {"x-request-id": "492a549c-2a07-4825-8f6f-4b000af3d72b", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
10882025-04-30T06:42:51.101ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_nexus_external
10892025-04-30T06:42:51.101ZDEBGtest_snapshot_expunge: Starting Nexus (external API)
10902025-04-30T06:42:54.329ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
10912025-04-30T06:42:54.329ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10922025-04-30T06:42:54.329ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
10932025-04-30T06:42:54.329ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
10942025-04-30T06:42:54.332ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10952025-04-30T06:42:54.334ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 4.482800196s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10962025-04-30T06:42:54.336ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10972025-04-30T06:42:54.343ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
10982025-04-30T06:42:54.343ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10992025-04-30T06:42:54.343ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11002025-04-30T06:42:54.373ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11012025-04-30T06:42:54.394ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
11022025-04-30T06:42:54.394ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11032025-04-30T06:42:54.395ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11042025-04-30T06:42:54.395ZTRACtest_snapshot_expunge (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
11052025-04-30T06:42:54.395ZINFOtest_snapshot_expunge (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11062025-04-30T06:42:54.395ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 4.549299401s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11072025-04-30T06:42:54.396ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11082025-04-30T06:42:54.396ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11092025-04-30T06:42:54.396ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11102025-04-30T06:42:54.396ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11112025-04-30T06:42:54.396ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11122025-04-30T06:42:54.397ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11132025-04-30T06:42:54.409ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
11142025-04-30T06:42:54.409ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
11152025-04-30T06:42:54.409ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
11162025-04-30T06:42:54.409ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
11172025-04-30T06:42:54.451ZTRACtest_snapshot_expunge (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
11182025-04-30T06:42:54.452ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 4.596747288s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
11192025-04-30T06:42:54.541ZINFOtest_snapshot_expunge (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
11202025-04-30T06:42:54.541ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:51119/vpc-routes
11212025-04-30T06:42:54.542ZWARNtest_snapshot_expunge (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
11222025-04-30T06:42:54.543ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 4.713083261s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
11232025-04-30T06:42:54.543ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11242025-04-30T06:42:54.543ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11252025-04-30T06:42:54.544ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11262025-04-30T06:42:54.544ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11272025-04-30T06:42:54.548ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:55886 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11282025-04-30T06:42:54.551ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:55886 req_id = 9ef00112-328e-404e-ac8f-79ead570dff3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11292025-04-30T06:42:54.558ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 7526 local_addr = [::1]:51119 method = GET remote_addr = [::1]:55886 req_id = 9ef00112-328e-404e-ac8f-79ead570dff3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11302025-04-30T06:42:54.559ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11312025-04-30T06:42:54.562ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11322025-04-30T06:42:54.562ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11332025-04-30T06:42:54.562ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
11342025-04-30T06:42:54.563ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11352025-04-30T06:42:54.565ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:51119/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ef00112-328e-404e-ac8f-79ead570dff3", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
11362025-04-30T06:42:54.566ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
11372025-04-30T06:42:54.566ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11382025-04-30T06:42:54.566ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11392025-04-30T06:42:54.775ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11402025-04-30T06:42:54.775ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11412025-04-30T06:42:54.775ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11422025-04-30T06:42:54.775ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11432025-04-30T06:42:54.777ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11442025-04-30T06:42:54.782ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 4.928754491s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11452025-04-30T06:42:54.782ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11462025-04-30T06:42:54.786ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11472025-04-30T06:42:54.787ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11482025-04-30T06:42:54.787ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11492025-04-30T06:42:54.787ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11502025-04-30T06:42:54.787ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11512025-04-30T06:42:54.787ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11522025-04-30T06:42:54.787ZINFOtest_snapshot_expunge (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
11532025-04-30T06:42:54.788ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 4.948962433s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11542025-04-30T06:42:54.792ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11552025-04-30T06:42:54.792ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11562025-04-30T06:42:54.800ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d1 a2 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55449 req_id = 4ddae148-7f9e-4e43-bba6-d2bea5e762dd
11572025-04-30T06:42:54.800ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53666,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55449 req_id = 4ddae148-7f9e-4e43-bba6-d2bea5e762dd
11582025-04-30T06:42:54.801ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
11592025-04-30T06:42:54.801ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11602025-04-30T06:42:54.801ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55449 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4ddae148-7f9e-4e43-bba6-d2bea5e762dd
11612025-04-30T06:42:54.803ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11622025-04-30T06:42:54.803ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
11632025-04-30T06:42:54.803ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11642025-04-30T06:42:54.805ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4239, tv_nsec: 604610449 } })
11652025-04-30T06:42:54.805ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
11662025-04-30T06:42:54.831ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11672025-04-30T06:42:54.835ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11682025-04-30T06:42:54.835ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11692025-04-30T06:42:54.835ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11702025-04-30T06:42:54.836ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11712025-04-30T06:42:54.836ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11722025-04-30T06:42:54.860ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11732025-04-30T06:42:54.860ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11742025-04-30T06:42:54.860ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11752025-04-30T06:42:54.860ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11762025-04-30T06:42:54.862ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11772025-04-30T06:42:54.862ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11782025-04-30T06:42:54.862ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11792025-04-30T06:42:54.863ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11802025-04-30T06:42:54.868ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 65 38 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e e8..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51135 req_id = 9c733345-0525-43b0-b287-000ce3b4b432
11812025-04-30T06:42:54.868ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25912,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51135 req_id = 9c733345-0525-43b0-b287-000ce3b4b432
11822025-04-30T06:42:54.869ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11832025-04-30T06:42:54.869ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51135 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9c733345-0525-43b0-b287-000ce3b4b432
11842025-04-30T06:42:54.870ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11852025-04-30T06:42:54.870ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11862025-04-30T06:42:54.870ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11872025-04-30T06:42:54.870ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11882025-04-30T06:42:54.871ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11892025-04-30T06:42:54.872ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11902025-04-30T06:42:54.887ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11912025-04-30T06:42:54.887ZTRACtest_snapshot_expunge (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11922025-04-30T06:42:54.888ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11932025-04-30T06:42:54.888ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11942025-04-30T06:42:54.888ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11952025-04-30T06:42:54.888ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11962025-04-30T06:42:54.888ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11972025-04-30T06:42:54.939ZINFOtest_snapshot_expunge (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
11982025-04-30T06:42:54.939ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:51119/eip-gateways
11992025-04-30T06:42:54.941ZINFOtest_snapshot_expunge (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
12002025-04-30T06:42:54.941ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12012025-04-30T06:42:54.941ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
12022025-04-30T06:42:54.941ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12032025-04-30T06:42:54.944ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = PUT remote_addr = [::1]:55886 req_id = 712fb5d3-286a-418a-8a11-510017c10c42 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12042025-04-30T06:42:55.051ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
12052025-04-30T06:42:55.052ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 108204 local_addr = [::1]:51119 method = PUT remote_addr = [::1]:55886 req_id = 712fb5d3-286a-418a-8a11-510017c10c42 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12062025-04-30T06:42:55.052ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12072025-04-30T06:42:55.060ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12082025-04-30T06:42:55.060ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12092025-04-30T06:42:55.060ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12102025-04-30T06:42:55.060ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12112025-04-30T06:42:55.062ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12122025-04-30T06:42:55.062ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12132025-04-30T06:42:55.062ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12142025-04-30T06:42:55.063ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12152025-04-30T06:42:55.072ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:51119/eip-gateways", status: 204, headers: {"x-request-id": "712fb5d3-286a-418a-8a11-510017c10c42", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
12162025-04-30T06:42:55.074ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 5.226148358s iteration = 1 status = {}
12172025-04-30T06:42:55.075ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:51119/artifacts-config
12182025-04-30T06:42:55.077ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12192025-04-30T06:42:55.077ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12202025-04-30T06:42:55.077ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12212025-04-30T06:42:55.077ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12222025-04-30T06:42:55.078ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12232025-04-30T06:42:55.078ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12242025-04-30T06:42:55.078ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12252025-04-30T06:42:55.078ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
12262025-04-30T06:42:55.081ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12272025-04-30T06:42:55.082ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:39404 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12282025-04-30T06:42:55.083ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 47 1d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c G............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58083 req_id = e197e399-52e1-48f8-a7eb-250fc9d03d18
12292025-04-30T06:42:55.083ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18205,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58083 req_id = e197e399-52e1-48f8-a7eb-250fc9d03d18
12302025-04-30T06:42:55.083ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
12312025-04-30T06:42:55.083ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18205, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58083 req_id = e197e399-52e1-48f8-a7eb-250fc9d03d18
12322025-04-30T06:42:55.086ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = PUT remote_addr = [::1]:39404 req_id = 310ce6e7-8767-4b48-be8b-fcfa2e4b6240 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12332025-04-30T06:42:55.086ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
12342025-04-30T06:42:55.086ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
12352025-04-30T06:42:55.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
12362025-04-30T06:42:55.096ZINFOtest_snapshot_expunge (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpTUreDe/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12372025-04-30T06:42:55.096ZDEBGtest_snapshot_expunge (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpTUreDe/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12382025-04-30T06:42:55.096ZINFOtest_snapshot_expunge (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpphPCl1/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12392025-04-30T06:42:55.096ZDEBGtest_snapshot_expunge (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpphPCl1/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12402025-04-30T06:42:55.096ZINFOtest_snapshot_expunge (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpTUreDe/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12412025-04-30T06:42:55.115ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12422025-04-30T06:42:55.126ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12432025-04-30T06:42:55.126ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12442025-04-30T06:42:55.126ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12452025-04-30T06:42:55.127ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12462025-04-30T06:42:55.207ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:51119/v2p
12472025-04-30T06:42:55.209ZINFOtest_snapshot_expunge (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpphPCl1/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12482025-04-30T06:42:55.211ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12492025-04-30T06:42:55.211ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12502025-04-30T06:42:55.212ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12512025-04-30T06:42:55.214ZINFOtest_snapshot_expunge (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12522025-04-30T06:42:55.214ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 128029 local_addr = [::1]:51119 method = PUT remote_addr = [::1]:39404 req_id = 310ce6e7-8767-4b48-be8b-fcfa2e4b6240 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12532025-04-30T06:42:55.215ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12542025-04-30T06:42:55.226ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12552025-04-30T06:42:55.226ZDEBGtest_snapshot_expunge (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12562025-04-30T06:42:55.226ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12572025-04-30T06:42:55.226ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
12582025-04-30T06:42:55.228ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12592025-04-30T06:42:55.231ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:51349 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12602025-04-30T06:42:55.231ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:51119/artifacts-config", status: 204, headers: {"x-request-id": "310ce6e7-8767-4b48-be8b-fcfa2e4b6240", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
12612025-04-30T06:42:55.231ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:51119/artifacts
12622025-04-30T06:42:55.232ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12632025-04-30T06:42:55.232ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12642025-04-30T06:42:55.233ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12652025-04-30T06:42:55.235ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:39404 req_id = d5490c06-5dc2-49f2-8e0a-d531adfea33c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12662025-04-30T06:42:55.235ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:51349 req_id = 9d292c39-d84c-46ba-8262-41fd31abf9fd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12672025-04-30T06:42:55.239ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 4465 local_addr = [::1]:51119 method = GET remote_addr = [::1]:39404 req_id = d5490c06-5dc2-49f2-8e0a-d531adfea33c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12682025-04-30T06:42:55.239ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 4051 local_addr = [::1]:51119 method = GET remote_addr = [::1]:51349 req_id = 9d292c39-d84c-46ba-8262-41fd31abf9fd response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12692025-04-30T06:42:55.241ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:51119/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5490c06-5dc2-49f2-8e0a-d531adfea33c", "content-length": "26", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
12702025-04-30T06:42:55.241ZINFOtest_snapshot_expunge (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:51119
12712025-04-30T06:42:55.242ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:51119/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d292c39-d84c-46ba-8262-41fd31abf9fd", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
12722025-04-30T06:42:55.242ZINFOtest_snapshot_expunge (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12732025-04-30T06:42:55.242ZINFOtest_snapshot_expunge (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12742025-04-30T06:42:55.242ZINFOtest_snapshot_expunge (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12752025-04-30T06:42:55.243ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 5.400372398s iteration = 1 status = {}
12762025-04-30T06:42:55.245ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 5.39060607s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:42:55.231643086Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:42:55.241797668Z"}]}
12772025-04-30T06:42:55.245ZINFOtest_snapshot_expunge (InternalApi): Using static configuration for external switchports
12782025-04-30T06:42:55.245ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12792025-04-30T06:42:55.245ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
12802025-04-30T06:42:55.245ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12812025-04-30T06:42:55.249ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12822025-04-30T06:42:55.249ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12832025-04-30T06:42:55.249ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12842025-04-30T06:42:55.262ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12852025-04-30T06:42:55.262ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12862025-04-30T06:42:55.262ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12872025-04-30T06:42:55.262ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12882025-04-30T06:42:55.263ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12892025-04-30T06:42:55.268ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab9b9855-7b04-4ccf-b56a-39ef6cc33fd4", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
12902025-04-30T06:42:55.269ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
12912025-04-30T06:42:55.294ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12922025-04-30T06:42:55.294ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12932025-04-30T06:42:55.294ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12942025-04-30T06:42:55.294ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12952025-04-30T06:42:55.300ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12962025-04-30T06:42:55.300ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12972025-04-30T06:42:55.300ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12982025-04-30T06:42:55.301ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12992025-04-30T06:42:55.307ZINFOtest_snapshot_expunge (InternalApi): Recording Rack Network Configuration
13002025-04-30T06:42:55.307ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13012025-04-30T06:42:55.307ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13022025-04-30T06:42:55.308ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13032025-04-30T06:42:55.309ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13042025-04-30T06:42:55.318ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13052025-04-30T06:42:55.318ZTRACtest_snapshot_expunge (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13062025-04-30T06:42:55.318ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13072025-04-30T06:42:55.319ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13082025-04-30T06:42:55.319ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13092025-04-30T06:42:55.332ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13102025-04-30T06:42:55.332ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13112025-04-30T06:42:55.332ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13122025-04-30T06:42:55.338ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0a4697d-bab7-4d49-a8a9-3706c9db6281", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
13132025-04-30T06:42:55.340ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
13142025-04-30T06:42:55.340ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13152025-04-30T06:42:55.340ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13162025-04-30T06:42:55.342ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13172025-04-30T06:42:55.365ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13182025-04-30T06:42:55.365ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13192025-04-30T06:42:55.366ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13202025-04-30T06:42:55.392ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13212025-04-30T06:42:55.392ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13222025-04-30T06:42:55.392ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13232025-04-30T06:42:55.392ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13242025-04-30T06:42:55.393ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13252025-04-30T06:42:55.407ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:42:55.408ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:42:55.408ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:42:55.408ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:42:55.410ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
13302025-04-30T06:42:55.410ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13312025-04-30T06:42:55.410ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13322025-04-30T06:42:55.410ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13332025-04-30T06:42:55.410ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13342025-04-30T06:42:55.429ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13352025-04-30T06:42:55.430ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13362025-04-30T06:42:55.430ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13372025-04-30T06:42:55.431ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13382025-04-30T06:42:55.442ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13392025-04-30T06:42:55.451ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13402025-04-30T06:42:55.451ZTRACtest_snapshot_expunge (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13412025-04-30T06:42:55.451ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13422025-04-30T06:42:55.452ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13432025-04-30T06:42:55.452ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13442025-04-30T06:42:55.452ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13452025-04-30T06:42:55.452ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13462025-04-30T06:42:55.453ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13472025-04-30T06:42:55.457ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
13482025-04-30T06:42:55.457ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13492025-04-30T06:42:55.457ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13502025-04-30T06:42:55.458ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13512025-04-30T06:42:55.460ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef538a2b-3d2a-48a0-8042-9f1dd10b9be7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
13522025-04-30T06:42:55.461ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
13532025-04-30T06:42:55.473ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13542025-04-30T06:42:55.474ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13552025-04-30T06:42:55.474ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13562025-04-30T06:42:55.475ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13572025-04-30T06:42:55.483ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13582025-04-30T06:42:55.483ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13592025-04-30T06:42:55.483ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13602025-04-30T06:42:55.484ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13612025-04-30T06:42:55.484ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: d881068f-66db-47f0-96d3-7b59850736d2, lookup_type: ByName("oxide-service-pool") }
13622025-04-30T06:42:55.484ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13632025-04-30T06:42:55.484ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13642025-04-30T06:42:55.484ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13652025-04-30T06:42:55.485ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13662025-04-30T06:42:55.495ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13672025-04-30T06:42:55.503ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13682025-04-30T06:42:55.504ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13692025-04-30T06:42:55.504ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13702025-04-30T06:42:55.504ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13712025-04-30T06:42:55.515ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13722025-04-30T06:42:55.515ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13732025-04-30T06:42:55.516ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13742025-04-30T06:42:55.516ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13752025-04-30T06:42:55.550ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: d881068f-66db-47f0-96d3-7b59850736d2, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13762025-04-30T06:42:55.550ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13772025-04-30T06:42:55.550ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13782025-04-30T06:42:55.550ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13792025-04-30T06:42:55.557ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13802025-04-30T06:42:55.557ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13812025-04-30T06:42:55.558ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13822025-04-30T06:42:55.572ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13832025-04-30T06:42:55.572ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13842025-04-30T06:42:55.573ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13852025-04-30T06:42:55.573ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: d881068f-66db-47f0-96d3-7b59850736d2, lookup_type: ByName("oxide-service-pool") }
13862025-04-30T06:42:55.573ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13872025-04-30T06:42:55.573ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13882025-04-30T06:42:55.573ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13892025-04-30T06:42:55.574ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13902025-04-30T06:42:55.576ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62a496c8-93e7-4a24-ba95-e717e6d23b82", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
13912025-04-30T06:42:55.576ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
13922025-04-30T06:42:55.585ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13932025-04-30T06:42:55.585ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13942025-04-30T06:42:55.586ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13952025-04-30T06:42:55.588ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13962025-04-30T06:42:55.594ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: d881068f-66db-47f0-96d3-7b59850736d2, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13972025-04-30T06:42:55.605ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13982025-04-30T06:42:55.605ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13992025-04-30T06:42:55.605ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14002025-04-30T06:42:55.605ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14012025-04-30T06:42:55.606ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14022025-04-30T06:42:55.606ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: d881068f-66db-47f0-96d3-7b59850736d2, lookup_type: ByName("oxide-service-pool") }
14032025-04-30T06:42:55.607ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14042025-04-30T06:42:55.607ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14052025-04-30T06:42:55.607ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
14062025-04-30T06:42:55.608ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14072025-04-30T06:42:55.612ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14082025-04-30T06:42:55.612ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14092025-04-30T06:42:55.612ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14102025-04-30T06:42:55.613ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14112025-04-30T06:42:55.616ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14122025-04-30T06:42:55.616ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14132025-04-30T06:42:55.616ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14142025-04-30T06:42:55.617ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14152025-04-30T06:42:55.618ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14162025-04-30T06:42:55.622ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: d881068f-66db-47f0-96d3-7b59850736d2, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14172025-04-30T06:42:55.629ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14182025-04-30T06:42:55.629ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14192025-04-30T06:42:55.629ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14202025-04-30T06:42:55.629ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
14212025-04-30T06:42:55.630ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14222025-04-30T06:42:55.632ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14232025-04-30T06:42:55.650ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14242025-04-30T06:42:55.650ZTRACtest_snapshot_expunge (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
14252025-04-30T06:42:55.650ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14262025-04-30T06:42:55.651ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14272025-04-30T06:42:55.651ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14282025-04-30T06:42:55.651ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14292025-04-30T06:42:55.651ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14302025-04-30T06:42:55.653ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14312025-04-30T06:42:55.658ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14322025-04-30T06:42:55.663ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14332025-04-30T06:42:55.663ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14342025-04-30T06:42:55.663ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14352025-04-30T06:42:55.664ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14362025-04-30T06:42:55.672ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14372025-04-30T06:42:55.672ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14382025-04-30T06:42:55.672ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14392025-04-30T06:42:55.673ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14402025-04-30T06:42:55.676ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8fef403-b0c0-4cbd-acd1-7a4681b181ec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14412025-04-30T06:42:55.676ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
14422025-04-30T06:42:55.679ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14432025-04-30T06:42:55.686ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14442025-04-30T06:42:55.687ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14452025-04-30T06:42:55.687ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14462025-04-30T06:42:55.687ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14472025-04-30T06:42:55.701ZDEBGtest_snapshot_expunge (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
14482025-04-30T06:42:55.702ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14492025-04-30T06:42:55.702ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14502025-04-30T06:42:55.702ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14512025-04-30T06:42:55.702ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14522025-04-30T06:42:55.702ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14532025-04-30T06:42:55.709ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14542025-04-30T06:42:55.714ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14552025-04-30T06:42:55.716ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14562025-04-30T06:42:55.716ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14572025-04-30T06:42:55.717ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14582025-04-30T06:42:55.738ZTRACtest_snapshot_expunge (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14592025-04-30T06:42:55.738ZDEBGtest_snapshot_expunge (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14602025-04-30T06:42:55.738ZDEBGtest_snapshot_expunge (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14612025-04-30T06:42:55.738ZDEBGtest_snapshot_expunge (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14622025-04-30T06:42:55.769ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa53befd-d164-4623-9db8-7e34b45a1beb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14632025-04-30T06:42:55.769ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
14642025-04-30T06:42:55.771ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55e673a5-661a-4bed-beec-431e9367410c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14652025-04-30T06:42:55.771ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
14662025-04-30T06:42:55.774ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfa0e843-0f1e-44f0-b430-8fd9f29cb748", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14672025-04-30T06:42:55.775ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
14682025-04-30T06:42:55.776ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81ab4cd1-288f-4623-9c30-71e0a137448a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14692025-04-30T06:42:55.776ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
14702025-04-30T06:42:55.777ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5900d2cf-24fd-48b5-9879-195fff55de37", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14712025-04-30T06:42:55.778ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
14722025-04-30T06:42:55.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5339744d-b5e7-46df-b993-894d54a1cb67", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14732025-04-30T06:42:55.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
14742025-04-30T06:42:55.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f05e489-e0b4-4e86-8f44-e9b959aeef7d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14752025-04-30T06:42:55.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
14762025-04-30T06:42:55.781ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3119e5c4-7404-4533-a118-3180ddd38264", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14772025-04-30T06:42:55.781ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
14782025-04-30T06:42:55.783ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aff50b67-18c1-4742-afda-b5761628d55a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14792025-04-30T06:42:55.784ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
14802025-04-30T06:42:55.791ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23505257-c86f-458b-9fe7-496ffdd9f3b6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14812025-04-30T06:42:55.791ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
14822025-04-30T06:42:55.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a83492e-25ea-4494-afb6-da0cd2f1aed9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14832025-04-30T06:42:55.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
14842025-04-30T06:42:55.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0671845b-ec6d-450e-9607-5e1b3c9b4992", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14852025-04-30T06:42:55.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
14862025-04-30T06:42:55.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43de9ae6-dc0d-43d5-a13b-e50933168574", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14872025-04-30T06:42:55.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
14882025-04-30T06:42:55.815ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83beb660-bd1a-41c5-8c50-558428e764d9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14892025-04-30T06:42:55.816ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
14902025-04-30T06:42:55.817ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a5a3bee-4d55-4059-b56e-b87a45a3f067", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14912025-04-30T06:42:55.818ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
14922025-04-30T06:42:55.819ZINFOtest_snapshot_expunge (InternalApi): inserted blueprint blueprint_id = 2be7fa2f-37f3-4827-abc6-93bcee22b135
14932025-04-30T06:42:55.819ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14942025-04-30T06:42:55.820ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14952025-04-30T06:42:55.820ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14962025-04-30T06:42:55.820ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
14972025-04-30T06:42:55.820ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14982025-04-30T06:42:55.822ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8c23095-b596-4c69-8f22-7791d7c42cc3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
14992025-04-30T06:42:55.822ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
15002025-04-30T06:42:55.823ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15012025-04-30T06:42:55.828ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15022025-04-30T06:42:55.829ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52187515-0162-4464-b6ac-5a21964fb6f9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15032025-04-30T06:42:55.829ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
15042025-04-30T06:42:55.831ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbdd8fcd-4413-4ed3-8480-b45c4685d917", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15052025-04-30T06:42:55.831ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
15062025-04-30T06:42:55.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fb67724-1165-4b9c-bf11-fec199e4132b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15072025-04-30T06:42:55.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
15082025-04-30T06:42:55.836ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67d8f62b-626d-4d9b-b1af-54cc783b5d19", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15092025-04-30T06:42:55.836ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
15102025-04-30T06:42:55.839ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58130e0c-3f9b-4df1-a156-0175ddfac5c8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15112025-04-30T06:42:55.839ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
15122025-04-30T06:42:55.842ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b7438a0-658f-447c-a0ce-39d2f4593955", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15132025-04-30T06:42:55.843ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
15142025-04-30T06:42:55.846ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f42f1f87-b060-4faa-92dc-aa0fb9ab5e8d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15152025-04-30T06:42:55.846ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
15162025-04-30T06:42:55.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7071e232-7c2e-4ffb-a7f1-6dc9b0a5d97e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15172025-04-30T06:42:55.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
15182025-04-30T06:42:55.853ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3d8ec7b-3923-40b8-9151-98a01786feb4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15192025-04-30T06:42:55.853ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
15202025-04-30T06:42:55.855ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e30339f1-ceb1-4ac1-bae0-c2346487ce0c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15212025-04-30T06:42:55.855ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
15222025-04-30T06:42:55.856ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0328ed0d-af6c-4a5c-87c5-dadef5e0b1a9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15232025-04-30T06:42:55.856ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
15242025-04-30T06:42:55.857ZDEBGtest_snapshot_expunge (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
15252025-04-30T06:42:55.857ZDEBGtest_snapshot_expunge (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
15262025-04-30T06:42:55.857ZINFOtest_snapshot_expunge (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
15272025-04-30T06:42:55.857ZDEBGtest_snapshot_expunge (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 6.010126423s
15282025-04-30T06:42:55.857ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 6.010259018s iteration = 1 status = {}
15292025-04-30T06:42:55.858ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cba1b609-50a7-4756-9c7f-1ea8adbee2c1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15302025-04-30T06:42:55.859ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
15312025-04-30T06:42:55.861ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5c34598-2d26-4b2c-926a-f0c5034d66ad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15322025-04-30T06:42:55.862ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
15332025-04-30T06:42:55.863ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10a341f6-2e07-49f6-be6c-00d5a0010b7b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15342025-04-30T06:42:55.864ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
15352025-04-30T06:42:55.867ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7604790-f596-49a8-b5f9-f46846fe1ae9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15362025-04-30T06:42:55.867ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
15372025-04-30T06:42:55.869ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d216c62f-2669-444d-b3f7-80905de78996", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15382025-04-30T06:42:55.869ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
15392025-04-30T06:42:55.871ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1014035-9d40-4cad-a9d9-e2ccd94b1daa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15402025-04-30T06:42:55.871ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
15412025-04-30T06:42:55.877ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdc2a9e5-2b6f-4c38-a744-e5e2c37efc1d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15422025-04-30T06:42:55.877ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
15432025-04-30T06:42:55.881ZINFOtest_snapshot_expunge (InternalApi): Inserted networking records for nexus service
15442025-04-30T06:42:55.882ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "450ed9c6-57a7-48fb-9da2-41a54b929a4c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15452025-04-30T06:42:55.883ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
15462025-04-30T06:42:55.894ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d74d7885-885c-43df-9f52-3cf69ad0cc48", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15472025-04-30T06:42:55.895ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
15482025-04-30T06:42:55.897ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11eb12c5-dd54-40f5-8595-550c12ae9e8b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15492025-04-30T06:42:55.897ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
15502025-04-30T06:42:55.898ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "601a8da2-6462-4248-b92d-30fd5f040072", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15512025-04-30T06:42:55.898ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
15522025-04-30T06:42:55.899ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cce2f63-a022-4403-b610-4657090db8ad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15532025-04-30T06:42:55.899ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
15542025-04-30T06:42:55.900ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b719947-09a1-4b78-8782-0d1f644afb73", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
15552025-04-30T06:42:55.900ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15562025-04-30T06:42:55.900ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15572025-04-30T06:42:55.900ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15582025-04-30T06:42:55.900ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15592025-04-30T06:42:55.901ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15602025-04-30T06:42:55.907ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15612025-04-30T06:42:55.913ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15622025-04-30T06:42:55.913ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15632025-04-30T06:42:55.913ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15642025-04-30T06:42:55.913ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15652025-04-30T06:42:55.913ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15662025-04-30T06:42:55.914ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15672025-04-30T06:42:55.918ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15682025-04-30T06:42:55.922ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15692025-04-30T06:42:55.922ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15702025-04-30T06:42:55.923ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15712025-04-30T06:42:55.924ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15722025-04-30T06:42:55.924ZINFOtest_snapshot_expunge (InternalApi): Inserted networking records for external_dns service
15732025-04-30T06:42:55.924ZINFOtest_snapshot_expunge (InternalApi): No networking records needed for clickhouse service
15742025-04-30T06:42:55.924ZINFOtest_snapshot_expunge (InternalApi): No networking records needed for cockroach_db service
15752025-04-30T06:42:55.924ZINFOtest_snapshot_expunge (InternalApi): No networking records needed for internal_dns service
15762025-04-30T06:42:55.924ZINFOtest_snapshot_expunge (InternalApi): No networking records needed for crucible_pantry service
15772025-04-30T06:42:55.924ZINFOtest_snapshot_expunge (InternalApi): Inserted service networking records
15782025-04-30T06:42:55.924ZINFOtest_snapshot_expunge (InternalApi): Inserted physical disks
15792025-04-30T06:42:55.924ZINFOtest_snapshot_expunge (InternalApi): Inserted zpools
15802025-04-30T06:42:55.925ZINFOtest_snapshot_expunge (InternalApi): Inserted datasets
15812025-04-30T06:42:55.946ZINFOtest_snapshot_expunge (InternalApi): Populated DNS tables for internal DNS
15822025-04-30T06:42:55.976ZINFOtest_snapshot_expunge (InternalApi): Populated DNS tables for external DNS
15832025-04-30T06:42:55.976ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
15842025-04-30T06:42:55.976ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15852025-04-30T06:42:55.977ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15862025-04-30T06:42:55.977ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
15872025-04-30T06:42:55.977ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15882025-04-30T06:42:55.979ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15892025-04-30T06:42:55.986ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
15902025-04-30T06:42:55.987ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
15912025-04-30T06:42:55.987ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15922025-04-30T06:42:55.987ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15932025-04-30T06:42:55.987ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
15942025-04-30T06:42:55.987ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15952025-04-30T06:42:55.991ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15962025-04-30T06:42:55.995ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
15972025-04-30T06:42:56.113ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
15982025-04-30T06:42:56.113ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
15992025-04-30T06:42:56.114ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16002025-04-30T06:42:56.114ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16012025-04-30T06:42:56.114ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16022025-04-30T06:42:56.114ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16032025-04-30T06:42:56.115ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16042025-04-30T06:42:56.115ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:40987 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
16052025-04-30T06:42:56.115ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:40987 req_id = 35fd8c01-d2d6-4599-ada2-b38c2aa7bb88 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
16062025-04-30T06:42:56.116ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 571 local_addr = [::1]:51119 method = GET remote_addr = [::1]:40987 req_id = 35fd8c01-d2d6-4599-ada2-b38c2aa7bb88 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
16072025-04-30T06:42:56.116ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "35fd8c01-d2d6-4599-ada2-b38c2aa7bb88", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
16082025-04-30T06:42:56.118ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
16092025-04-30T06:42:56.118ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
16102025-04-30T06:42:56.118ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
16112025-04-30T06:42:56.119ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
16122025-04-30T06:42:56.119ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16132025-04-30T06:42:56.119ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
16142025-04-30T06:42:56.119ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16152025-04-30T06:42:56.120ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16162025-04-30T06:42:56.120ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16172025-04-30T06:42:56.127ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16182025-04-30T06:42:56.127ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16192025-04-30T06:42:56.127ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16202025-04-30T06:42:56.127ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16212025-04-30T06:42:56.127ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16222025-04-30T06:42:56.127ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16232025-04-30T06:42:56.130ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16242025-04-30T06:42:56.133ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
16252025-04-30T06:42:56.134ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
16262025-04-30T06:42:56.134ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16272025-04-30T06:42:56.134ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16282025-04-30T06:42:56.135ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16292025-04-30T06:42:56.142ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16302025-04-30T06:42:56.147ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16312025-04-30T06:42:56.148ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322025-04-30T06:42:56.148ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16332025-04-30T06:42:56.148ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16342025-04-30T06:42:56.148ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16352025-04-30T06:42:56.153ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362025-04-30T06:42:56.162ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16372025-04-30T06:42:56.189ZINFOtest_snapshot_expunge (InternalApi): Created recovery silo
16382025-04-30T06:42:56.192ZINFOtest_snapshot_expunge (InternalApi): Created recovery user
16392025-04-30T06:42:56.199ZINFOtest_snapshot_expunge (InternalApi): Created recovery user's password
16402025-04-30T06:42:56.199ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 064d601f-0a53-448d-b924-c528a8e9e0e8, lookup_type: ById(064d601f-0a53-448d-b924-c528a8e9e0e8) }
16412025-04-30T06:42:56.199ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 064d601f-0a53-448d-b924-c528a8e9e0e8 resource_type = Silo
16422025-04-30T06:42:56.199ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:42:56.199ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:42:56.200ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:42:56.208ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:42:56.208ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:42:56.208ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:42:56.209ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:42:56.220ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:42:56.222ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 064d601f-0a53-448d-b924-c528a8e9e0e8, lookup_type: ById(064d601f-0a53-448d-b924-c528a8e9e0e8) } result = Ok(())
16512025-04-30T06:42:56.222ZDEBGtest_snapshot_expunge (InternalApi): Generated role assignment queries
16522025-04-30T06:42:56.231ZINFOtest_snapshot_expunge (InternalApi): Granted Silo privileges
16532025-04-30T06:42:56.231ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:42:56.231ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:42:56.231ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:42:56.231ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:42:56.234ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:42:56.239ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:42:56.241ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:42:56.253ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:42:56.253ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:42:56.253ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:42:56.259ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:42:56.259ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:42:56.261ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:42:56.266ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:42:56.266ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:42:56.266ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:42:56.268ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:42:56.268ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:42:56.268ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:42:56.268ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:42:56.269ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:42:56.273ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:42:56.273ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:42:56.273ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:42:56.274ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:42:56.281ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:42:56.281ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:42:56.281ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:42:56.281ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:42:56.287ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:42:56.305ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:42:56.305ZTRACtest_snapshot_expunge (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:42:56.305ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:42:56.305ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:42:56.305ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:42:56.305ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:42:56.306ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:42:56.319ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:42:56.319ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:42:56.319ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:42:56.319ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:42:56.334ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16952025-04-30T06:42:56.335ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16962025-04-30T06:42:56.335ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16972025-04-30T06:42:56.335ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16982025-04-30T06:42:56.347ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-04-30T06:42:56.353ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17002025-04-30T06:42:56.354ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17012025-04-30T06:42:56.354ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17022025-04-30T06:42:56.354ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17032025-04-30T06:42:56.366ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:42:56.367ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:42:56.367ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:42:56.371ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17072025-04-30T06:42:56.372ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17082025-04-30T06:42:56.373ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17092025-04-30T06:42:56.376ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17102025-04-30T06:42:56.376ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17112025-04-30T06:42:56.377ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17122025-04-30T06:42:56.382ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17132025-04-30T06:42:56.382ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17142025-04-30T06:42:56.382ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:42:56.382ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:42:56.385ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:42:56.388ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17182025-04-30T06:42:56.388ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17192025-04-30T06:42:56.388ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17202025-04-30T06:42:56.388ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17212025-04-30T06:42:56.394ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222025-04-30T06:42:56.395ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:42:56.395ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:42:56.395ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:42:56.397ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-04-30T06:42:56.403ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17272025-04-30T06:42:56.403ZTRACtest_snapshot_expunge (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17282025-04-30T06:42:56.403ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17292025-04-30T06:42:56.403ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17302025-04-30T06:42:56.403ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17312025-04-30T06:42:56.405ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17322025-04-30T06:42:56.405ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17332025-04-30T06:42:56.407ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17342025-04-30T06:42:56.409ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17352025-04-30T06:42:56.409ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17362025-04-30T06:42:56.410ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17372025-04-30T06:42:56.414ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17382025-04-30T06:42:56.414ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17392025-04-30T06:42:56.414ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17402025-04-30T06:42:56.417ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17412025-04-30T06:42:56.417ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17422025-04-30T06:42:56.417ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:42:56.418ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:42:56.418ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:42:56.420ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17462025-04-30T06:42:56.420ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17472025-04-30T06:42:56.420ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17482025-04-30T06:42:56.421ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17492025-04-30T06:42:56.425ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17502025-04-30T06:42:56.425ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17512025-04-30T06:42:56.425ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17522025-04-30T06:42:56.425ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17532025-04-30T06:42:56.429ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17542025-04-30T06:42:56.439ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17552025-04-30T06:42:56.439ZTRACtest_snapshot_expunge (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17562025-04-30T06:42:56.440ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17572025-04-30T06:42:56.440ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17582025-04-30T06:42:56.440ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17592025-04-30T06:42:56.440ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17602025-04-30T06:42:56.440ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17612025-04-30T06:42:56.447ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17622025-04-30T06:42:56.447ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17632025-04-30T06:42:56.447ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17642025-04-30T06:42:56.448ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17652025-04-30T06:42:56.457ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17662025-04-30T06:42:56.457ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17672025-04-30T06:42:56.457ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17682025-04-30T06:42:56.458ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17692025-04-30T06:42:56.462ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17702025-04-30T06:42:56.472ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17712025-04-30T06:42:56.472ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17722025-04-30T06:42:56.472ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17732025-04-30T06:42:56.473ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17742025-04-30T06:42:56.480ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:42:56.480ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:42:56.480ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:42:56.484ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17782025-04-30T06:42:56.484ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17792025-04-30T06:42:56.485ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17802025-04-30T06:42:56.487ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17812025-04-30T06:42:56.487ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17822025-04-30T06:42:56.488ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17832025-04-30T06:42:56.492ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17842025-04-30T06:42:56.492ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17852025-04-30T06:42:56.494ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17862025-04-30T06:42:56.497ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17872025-04-30T06:42:56.497ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17882025-04-30T06:42:56.498ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17892025-04-30T06:42:56.498ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17902025-04-30T06:42:56.498ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17912025-04-30T06:42:56.501ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17922025-04-30T06:42:56.501ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17932025-04-30T06:42:56.501ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17942025-04-30T06:42:56.503ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17952025-04-30T06:42:56.507ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17962025-04-30T06:42:56.507ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:42:56.507ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:42:56.507ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:42:56.509ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18002025-04-30T06:42:56.515ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18012025-04-30T06:42:56.515ZTRACtest_snapshot_expunge (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18022025-04-30T06:42:56.515ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
18032025-04-30T06:42:56.515ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18042025-04-30T06:42:56.515ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:42:56.515ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:42:56.516ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:42:56.517ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18082025-04-30T06:42:56.518ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18092025-04-30T06:42:56.518ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18102025-04-30T06:42:56.518ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18112025-04-30T06:42:56.520ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18122025-04-30T06:42:56.520ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:42:56.520ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:42:56.520ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:42:56.522ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18162025-04-30T06:42:56.529ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18172025-04-30T06:42:56.529ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18182025-04-30T06:42:56.530ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18192025-04-30T06:42:56.530ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18202025-04-30T06:42:56.537ZDEBGtest_snapshot_expunge (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: b93f313f-d062-4189-939b-bc4fddea0705, kind: Service { id: 69aed484-7768-456c-9a68-bca265ad4317 }, name: Name("external-dns-69aed484-7768-456c-9a68-bca265ad4317"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 736df47b-f45d-4c92-840a-837099043a36, kind: Service { id: 58f2365f-b720-4ec1-ad50-a68fadcacb1c }, name: Name("nexus-58f2365f-b720-4ec1-ad50-a68fadcacb1c"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
18212025-04-30T06:42:56.537ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18222025-04-30T06:42:56.538ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18232025-04-30T06:42:56.538ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18242025-04-30T06:42:56.538ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18252025-04-30T06:42:56.540ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18262025-04-30T06:42:56.540ZINFOtest_snapshot_expunge (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = b9eba9a5-2952-4b7f-9d32-4296e0270f0a
18272025-04-30T06:42:56.540ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = b9eba9a5-2952-4b7f-9d32-4296e0270f0a time_started = 2025-04-30 06:42:55.086277 UTC
18282025-04-30T06:42:56.540ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 5.626867005s iteration = 2 status = {"collection_id":"b9eba9a5-2952-4b7f-9d32-4296e0270f0a","time_done":"2025-04-30 06:42:56.118314 UTC","time_started":"2025-04-30 06:42:55.086277 UTC"}
18292025-04-30T06:42:56.540ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
18302025-04-30T06:42:56.540ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 21.022\u{b5}s iteration = 2 status = {"error":"task disabled"}
18312025-04-30T06:42:56.540ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
18322025-04-30T06:42:56.540ZWARNtest_snapshot_expunge (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
18332025-04-30T06:42:56.540ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 15.301\u{b5}s iteration = 2 status = {"error":"no blueprint"}
18342025-04-30T06:42:56.542ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18352025-04-30T06:42:56.545ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18362025-04-30T06:42:56.545ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18372025-04-30T06:42:56.545ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18382025-04-30T06:42:56.546ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18392025-04-30T06:42:56.548ZTRACtest_snapshot_expunge (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:42:56.241733Z, allowed_ips: None }
18402025-04-30T06:42:56.548ZDEBGtest_snapshot_expunge (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:42:56.549ZDEBGtest_snapshot_expunge (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: b93f313f-d062-4189-939b-bc4fddea0705, kind: Service { id: 69aed484-7768-456c-9a68-bca265ad4317 }, name: Name("external-dns-69aed484-7768-456c-9a68-bca265ad4317"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 736df47b-f45d-4c92-840a-837099043a36, kind: Service { id: 58f2365f-b720-4ec1-ad50-a68fadcacb1c }, name: Name("nexus-58f2365f-b720-4ec1-ad50-a68fadcacb1c"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:42:56.549ZDEBGtest_snapshot_expunge (nexus): resolved 2 rules for sleds
18432025-04-30T06:42:56.638ZDEBGtest_snapshot_expunge (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:42:49.972333Z, time_modified: 2025-04-30T06:42:49.972333Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(51119), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(42164) }]
18442025-04-30T06:42:56.639ZDEBGtest_snapshot_expunge (nexus): sending firewall rules to sled agents
18452025-04-30T06:42:56.639ZTRACtest_snapshot_expunge (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:42:56.639ZDEBGtest_snapshot_expunge (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:42:56.639ZDEBGtest_snapshot_expunge (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:42:56.644ZTRACtest_snapshot_expunge (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:42:56.644ZTRACtest_snapshot_expunge (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:42:56.644ZTRACtest_snapshot_expunge (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:42:56.644ZDEBGtest_snapshot_expunge (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:42:56.645ZDEBGtest_snapshot_expunge (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:42:56.650ZDEBGtest_snapshot_expunge (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:42:56.658ZDEBGtest_snapshot_expunge (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:42:56.718ZDEBGtest_snapshot_expunge (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:51119/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:42:56.719ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:60542 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:42:56.719ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = PUT remote_addr = [::1]:60542 req_id = 04c408d4-4400-4129-bf4d-c36c9dbcd9d0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:42:56.721ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 1407 local_addr = [::1]:51119 method = PUT remote_addr = [::1]:60542 req_id = 04c408d4-4400-4129-bf4d-c36c9dbcd9d0 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:42:56.721ZDEBGtest_snapshot_expunge (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:51119/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "04c408d4-4400-4129-bf4d-c36c9dbcd9d0", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
18602025-04-30T06:42:56.725ZINFOtest_snapshot_expunge (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:42:56.725ZTRACtest_snapshot_expunge (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:42:56.725ZDEBGtest_snapshot_expunge (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:42:56.728ZDEBGtest_snapshot_expunge (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:42:56.728ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:42:56.728ZDEBGtest_snapshot_expunge (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:42:56.728ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:42:56.728ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:42:56.730ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:42:56.730ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:42:56.730ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:42:56.730ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:42:56.731ZDEBGtest_snapshot_expunge (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:42:56.731ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:42:56.731ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:42:56.731ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:42:56.731ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:42:56.731ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:42:56.732ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:42:56.732ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:42:56.732ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:42:56.732ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:42:56.732ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:42:56.732ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:42:56.732ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:42:56.732ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:42:56.733ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:42:56.733ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:42:56.733ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:42:56.734ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:42:56.734ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: cc 3e 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 .>..........._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:52078 req_id = e2ca28c8-bd8c-4d0b-a071-51cc8d5e8398
18912025-04-30T06:42:56.735ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52286,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52078 req_id = e2ca28c8-bd8c-4d0b-a071-51cc8d5e8398
18922025-04-30T06:42:56.735ZDEBGtest_snapshot_expunge (store): query key key = _nameservice._tcp
18932025-04-30T06:42:56.769ZDEBGtest_snapshot_expunge (store): query key key = 8fa73ed5-840a-435c-8250-4547cafe1025.host
18942025-04-30T06:42:56.769ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52078 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 58975, target: Name("8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal") })) }] req_id = e2ca28c8-bd8c-4d0b-a071-51cc8d5e8398
18952025-04-30T06:42:56.769ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 3a 63 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 :c..........._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49490 req_id = 37bde57b-939f-461c-8760-1933c464152f
18962025-04-30T06:42:56.769ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14947,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49490 req_id = 37bde57b-939f-461c-8760-1933c464152f
18972025-04-30T06:42:56.769ZDEBGtest_snapshot_expunge (store): query key key = _external-dns._tcp
18982025-04-30T06:42:56.769ZDEBGtest_snapshot_expunge (store): query key key = 69aed484-7768-456c-9a68-bca265ad4317.host
18992025-04-30T06:42:56.769ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49490 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46413, target: Name("69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal") })) }] req_id = 37bde57b-939f-461c-8760-1933c464152f
19002025-04-30T06:42:56.769ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 58975, target: Name("8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 537756757 } })
19012025-04-30T06:42:56.769ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
19022025-04-30T06:42:56.769ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46413, target: Name("69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 538054130 } })
19032025-04-30T06:42:56.769ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
19042025-04-30T06:42:56.769ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19052025-04-30T06:42:56.769ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19062025-04-30T06:42:56.769ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19072025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19082025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19092025-04-30T06:42:56.770ZTRACtest_snapshot_expunge (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19102025-04-30T06:42:56.770ZTRACtest_snapshot_expunge (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19112025-04-30T06:42:56.770ZTRACtest_snapshot_expunge (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19122025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (ServiceBalancer): roles roles = RoleSet { roles: {} }
19132025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19142025-04-30T06:42:56.770ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 02 61 01 00 00 01 00 00 00 00 00 01 24 38 66 61 .a..........$8fa\n0010: 37 33 65 64 35 2d 38 34 30 61 2d 34 33 35 63 2d 73ed5-840a-435c-\n0020: 38 32 35 30 2d 34 35 34 37 63 61 66 65 31 30 32 8250-4547cafe102\n0030: 35 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 5.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:59211 req_id = 893606e3-8029-47c4-afe2-66c326120e40
19152025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 609,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 56,\n 102,\n 97,\n 55,\n 51,\n 101,\n 100,\n 53,\n 45,\n 56,\n 52,\n 48,\n 97,\n 45,\n 52,\n 51,\n 53,\n 99,\n 45,\n 56,\n 50,\n 53,\n 48,\n 45,\n 52,\n 53,\n 52,\n 55,\n 99,\n 97,\n 102,\n 101,\n 49,\n 48,\n 50,\n 53,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59211 req_id = 893606e3-8029-47c4-afe2-66c326120e40
19162025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (store): query key key = 8fa73ed5-840a-435c-8250-4547cafe1025.host
19172025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59211 query = LowerQuery { name: LowerName(Name("8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal.")), original: Query { name: Name("8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("8fa73ed5-840a-435c-8250-4547cafe1025.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 893606e3-8029-47c4-afe2-66c326120e40
19182025-04-30T06:42:56.770ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 10 9a 01 00 00 01 00 00 00 00 00 01 24 36 39 61 ............$69a\n0010: 65 64 34 38 34 2d 37 37 36 38 2d 34 35 36 63 2d ed484-7768-456c-\n0020: 39 61 36 38 2d 62 63 61 32 36 35 61 64 34 33 31 9a68-bca265ad431\n0030: 37 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 7.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:60580 req_id = 99cc2e2d-7bab-400a-8d37-003e6f87a5b4
19192025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4250,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 54,\n 57,\n 97,\n 101,\n 100,\n 52,\n 56,\n 52,\n 45,\n 55,\n 55,\n 54,\n 56,\n 45,\n 52,\n 53,\n 54,\n 99,\n 45,\n 57,\n 97,\n 54,\n 56,\n 45,\n 98,\n 99,\n 97,\n 50,\n 54,\n 53,\n 97,\n 100,\n 52,\n 51,\n 49,\n 55,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60580 req_id = 99cc2e2d-7bab-400a-8d37-003e6f87a5b4
19202025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (store): query key key = 69aed484-7768-456c-9a68-bca265ad4317.host
19212025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60580 query = LowerQuery { name: LowerName(Name("69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal.")), original: Query { name: Name("69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("69aed484-7768-456c-9a68-bca265ad4317.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 99cc2e2d-7bab-400a-8d37-003e6f87a5b4
19222025-04-30T06:42:56.770ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19232025-04-30T06:42:56.770ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19242025-04-30T06:42:56.770ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19252025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19262025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19272025-04-30T06:42:56.770ZINFOtest_snapshot_expunge (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:58975] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 12.117474ms iteration = 2 status = {"addresses":["[::1]:58975"]}
19292025-04-30T06:42:56.770ZINFOtest_snapshot_expunge (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:46413] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 10.856391ms iteration = 2 status = {"addresses":["[::1]:46413"]}
19312025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:42:56.770ZWARNtest_snapshot_expunge (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 21.813\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:42:56.770ZWARNtest_snapshot_expunge (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:42:56.770ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 11.122\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19392025-04-30T06:42:56.771ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19402025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19412025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19422025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19442025-04-30T06:42:56.771ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19452025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19462025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19472025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19482025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19492025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19502025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19512025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19522025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19532025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19542025-04-30T06:42:56.771ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19552025-04-30T06:42:56.771ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19562025-04-30T06:42:56.771ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19572025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19582025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19592025-04-30T06:42:56.771ZDEBGtest_snapshot_expunge (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19602025-04-30T06:42:56.777ZDEBGtest_snapshot_expunge (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
19612025-04-30T06:42:56.777ZINFOtest_snapshot_expunge (nexus): Rack initialized
19622025-04-30T06:42:56.777ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19632025-04-30T06:42:56.777ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
19642025-04-30T06:42:56.777ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19652025-04-30T06:42:56.778ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19662025-04-30T06:42:56.783ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19672025-04-30T06:42:56.783ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19682025-04-30T06:42:56.784ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19692025-04-30T06:42:56.784ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19702025-04-30T06:42:56.786ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19712025-04-30T06:42:56.786ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
19722025-04-30T06:42:56.787ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19732025-04-30T06:42:56.788ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
19742025-04-30T06:42:56.788ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19752025-04-30T06:42:56.789ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ff d4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59027 req_id = 30ebc33c-9dcf-4398-8e29-747399296a5e
19762025-04-30T06:42:56.789ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65492,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59027 req_id = 30ebc33c-9dcf-4398-8e29-747399296a5e
19772025-04-30T06:42:56.790ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
19782025-04-30T06:42:56.790ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19792025-04-30T06:42:56.790ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59027 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 30ebc33c-9dcf-4398-8e29-747399296a5e
19802025-04-30T06:42:56.791ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 590197762 } })
19812025-04-30T06:42:56.791ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
19822025-04-30T06:42:56.791ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19832025-04-30T06:42:56.792ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
19842025-04-30T06:42:56.792ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19852025-04-30T06:42:56.793ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c5 47 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .G..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50211 req_id = c52c26c3-1fcb-4298-8313-9970cdcc0cfc
19862025-04-30T06:42:56.793ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50503,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50211 req_id = c52c26c3-1fcb-4298-8313-9970cdcc0cfc
19872025-04-30T06:42:56.793ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19882025-04-30T06:42:56.794ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50211 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c52c26c3-1fcb-4298-8313-9970cdcc0cfc
19892025-04-30T06:42:56.795ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
19902025-04-30T06:42:56.795ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
19912025-04-30T06:42:56.795ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19922025-04-30T06:42:56.795ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
19932025-04-30T06:42:56.796ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19942025-04-30T06:42:56.881ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
19952025-04-30T06:42:56.882ZDEBGtest_snapshot_expunge (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19962025-04-30T06:42:56.882ZDEBGtest_snapshot_expunge (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19972025-04-30T06:42:56.882ZDEBGtest_snapshot_expunge (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19982025-04-30T06:42:56.882ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19992025-04-30T06:42:56.882ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20002025-04-30T06:42:56.882ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20012025-04-30T06:42:56.882ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20022025-04-30T06:42:56.883ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20032025-04-30T06:42:56.883ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
20042025-04-30T06:42:56.883ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20052025-04-30T06:42:56.883ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20062025-04-30T06:42:56.883ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20072025-04-30T06:42:56.884ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20082025-04-30T06:42:56.885ZDEBGtest_snapshot_expunge (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
20092025-04-30T06:42:56.885ZDEBGtest_snapshot_expunge (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 3620797f-3983-4c88-a398-59d232909c53 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
20102025-04-30T06:42:56.885ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
20112025-04-30T06:42:56.885ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20122025-04-30T06:42:56.885ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20132025-04-30T06:42:56.885ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20142025-04-30T06:42:56.886ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20152025-04-30T06:42:56.887ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20162025-04-30T06:42:56.892ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20172025-04-30T06:42:56.892ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20182025-04-30T06:42:56.892ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20192025-04-30T06:42:56.893ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20202025-04-30T06:42:56.894ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20212025-04-30T06:42:56.899ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
20222025-04-30T06:42:56.899ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20232025-04-30T06:42:56.899ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20242025-04-30T06:42:56.900ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20252025-04-30T06:42:56.900ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20262025-04-30T06:42:56.907ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20272025-04-30T06:42:56.907ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20282025-04-30T06:42:56.907ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20292025-04-30T06:42:56.907ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20302025-04-30T06:42:56.909ZDEBGtest_snapshot_expunge (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20312025-04-30T06:42:56.909ZDEBGtest_snapshot_expunge (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = f55c679b-85eb-49d7-9bad-52e93b31f95c dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
20322025-04-30T06:42:56.909ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
20332025-04-30T06:42:56.909ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20342025-04-30T06:42:56.909ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20352025-04-30T06:42:56.909ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20362025-04-30T06:42:56.910ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20372025-04-30T06:42:56.911ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20382025-04-30T06:42:56.911ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20392025-04-30T06:42:56.911ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20402025-04-30T06:42:56.911ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20412025-04-30T06:42:56.911ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20422025-04-30T06:42:56.913ZDEBGtest_snapshot_expunge (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 3620797f-3983-4c88-a398-59d232909c53 dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20432025-04-30T06:42:56.913ZDEBGtest_snapshot_expunge (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20442025-04-30T06:42:56.913ZINFOtest_snapshot_expunge (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20452025-04-30T06:42:56.913ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 184.797076ms iteration = 2 status = {"generation":1}
20462025-04-30T06:42:56.913ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20472025-04-30T06:42:56.965ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:58975 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:58975] } uri = http://[::1]:58975/config
20482025-04-30T06:42:56.965ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20492025-04-30T06:42:56.972ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20502025-04-30T06:42:56.972ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20512025-04-30T06:42:56.972ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20522025-04-30T06:42:56.972ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20532025-04-30T06:42:56.973ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20542025-04-30T06:42:56.973ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20552025-04-30T06:42:56.973ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
20562025-04-30T06:42:56.973ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20572025-04-30T06:42:56.973ZINFOtest_snapshot_expunge (http): accepted connection kind = dns local_addr = [::1]:58975 remote_addr = [::1]:55036
20582025-04-30T06:42:56.974ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 64 9b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c d............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60289 req_id = ea74f645-b85c-472a-a78d-119a275ef33d
20592025-04-30T06:42:56.974ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25755,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60289 req_id = ea74f645-b85c-472a-a78d-119a275ef33d
20602025-04-30T06:42:56.974ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
20612025-04-30T06:42:56.974ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25755, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60289 req_id = ea74f645-b85c-472a-a78d-119a275ef33d
20622025-04-30T06:42:56.974ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20632025-04-30T06:42:56.978ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20642025-04-30T06:42:56.978ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20652025-04-30T06:42:56.978ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20662025-04-30T06:42:56.978ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20672025-04-30T06:42:56.979ZDEBGtest_snapshot_expunge (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = f55c679b-85eb-49d7-9bad-52e93b31f95c dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20682025-04-30T06:42:56.979ZDEBGtest_snapshot_expunge (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20692025-04-30T06:42:56.980ZINFOtest_snapshot_expunge (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20702025-04-30T06:42:56.980ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 248.713225ms iteration = 2 status = {"generation":2}
20712025-04-30T06:42:56.980ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20722025-04-30T06:42:57.039ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:46413 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:46413] } uri = http://[::1]:46413/config
20732025-04-30T06:42:57.041ZINFOtest_snapshot_expunge (http): accepted connection kind = dns local_addr = [::1]:46413 remote_addr = [::1]:51371
20742025-04-30T06:42:57.041ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
20752025-04-30T06:42:57.041ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
20762025-04-30T06:42:57.041ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
20772025-04-30T06:42:57.041ZTRACtest_snapshot_expunge (http): incoming request kind = dns local_addr = [::1]:58975 method = PUT remote_addr = [::1]:55036 req_id = 37f51087-0eae-4b3e-90f9-fbebd1682d09 uri = /config
20782025-04-30T06:42:57.042ZINFOtest_snapshot_expunge (store): attempting generation update new_generation = 1 req_id = 37f51087-0eae-4b3e-90f9-fbebd1682d09
20792025-04-30T06:42:57.042ZINFOtest_snapshot_expunge (store): updated generation new_generation = 1 req_id = 37f51087-0eae-4b3e-90f9-fbebd1682d09
20802025-04-30T06:42:57.043ZINFOtest_snapshot_expunge (http): request completed kind = dns latency_us = 1156 local_addr = [::1]:58975 method = PUT remote_addr = [::1]:55036 req_id = 37f51087-0eae-4b3e-90f9-fbebd1682d09 response_code = 204 uri = /config
20812025-04-30T06:42:57.043ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20822025-04-30T06:42:57.043ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20832025-04-30T06:42:57.043ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
20842025-04-30T06:42:57.044ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20852025-04-30T06:42:57.044ZTRACtest_snapshot_expunge (http): incoming request kind = dns local_addr = [::1]:46413 method = PUT remote_addr = [::1]:51371 req_id = 5fd4049f-f55d-4d0b-bc44-3fd80d3f8291 uri = /config
20862025-04-30T06:42:57.045ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:58975 generation = 1 result = Ok(Response { url: "http://[::1]:58975/config", status: 204, headers: {"x-request-id": "37f51087-0eae-4b3e-90f9-fbebd1682d09", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} }) servers = DnsServersList { addresses: [[::1]:58975] }
20872025-04-30T06:42:57.045ZINFOtest_snapshot_expunge (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:58975 generation = 1 servers = DnsServersList { addresses: [[::1]:58975] }
20882025-04-30T06:42:57.047ZINFOtest_snapshot_expunge (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:58975] }
20892025-04-30T06:42:57.047ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 134.205626ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:58975":{"Ok":null}}}
20902025-04-30T06:42:57.047ZINFOtest_snapshot_expunge (store): attempting generation update new_generation = 2 req_id = 5fd4049f-f55d-4d0b-bc44-3fd80d3f8291
20912025-04-30T06:42:57.047ZINFOtest_snapshot_expunge (store): pruning trees for generations newer than 0
20922025-04-30T06:42:57.047ZDEBGtest_snapshot_expunge (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20932025-04-30T06:42:57.048ZDEBGtest_snapshot_expunge (store): updating current config
20942025-04-30T06:42:57.048ZDEBGtest_snapshot_expunge (store): flushing default tree
20952025-04-30T06:42:57.048ZINFOtest_snapshot_expunge (store): pruning trees for generations older than 2 keep = 3
20962025-04-30T06:42:57.048ZINFOtest_snapshot_expunge (store): updated generation new_generation = 2 req_id = 5fd4049f-f55d-4d0b-bc44-3fd80d3f8291
20972025-04-30T06:42:57.048ZINFOtest_snapshot_expunge (http): request completed kind = dns latency_us = 4004 local_addr = [::1]:46413 method = PUT remote_addr = [::1]:51371 req_id = 5fd4049f-f55d-4d0b-bc44-3fd80d3f8291 response_code = 204 uri = /config
20982025-04-30T06:42:57.049ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:46413 generation = 2 result = Ok(Response { url: "http://[::1]:46413/config", status: 204, headers: {"x-request-id": "5fd4049f-f55d-4d0b-bc44-3fd80d3f8291", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} }) servers = DnsServersList { addresses: [[::1]:46413] }
20992025-04-30T06:42:57.049ZINFOtest_snapshot_expunge (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:46413 generation = 2 servers = DnsServersList { addresses: [[::1]:46413] }
21002025-04-30T06:42:57.051ZINFOtest_snapshot_expunge (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:46413] }
21012025-04-30T06:42:57.051ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 71.692553ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:46413":{"Ok":null}}}
21022025-04-30T06:42:57.053ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21032025-04-30T06:42:57.062ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21042025-04-30T06:42:57.062ZTRACtest_snapshot_expunge (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21052025-04-30T06:42:57.062ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21062025-04-30T06:42:57.062ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21072025-04-30T06:42:57.062ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21082025-04-30T06:42:57.062ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21092025-04-30T06:42:57.063ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21102025-04-30T06:42:57.082ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21112025-04-30T06:42:57.082ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21122025-04-30T06:42:57.082ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21132025-04-30T06:42:57.083ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21142025-04-30T06:42:57.084ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8679fe2-3e64-43f3-82ac-ad183255fcae", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21152025-04-30T06:42:57.084ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
21162025-04-30T06:42:57.092ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21172025-04-30T06:42:57.092ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21182025-04-30T06:42:57.092ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21192025-04-30T06:42:57.092ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21202025-04-30T06:42:57.104ZINFOtest_snapshot_expunge (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:42:48.015891Z, time_modified: 2025-04-30T06:42:48.015891Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 064d601f-0a53-448d-b924-c528a8e9e0e8, db_silo: Silo { identity: SiloIdentity { id: 064d601f-0a53-448d-b924-c528a8e9e0e8, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:42:55.976634Z, time_modified: 2025-04-30T06:42:55.976634Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: 064d601f-0a53-448d-b924-c528a8e9e0e8, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 064d601f-0a53-448d-b924-c528a8e9e0e8, db_silo: Silo { identity: SiloIdentity { id: 064d601f-0a53-448d-b924-c528a8e9e0e8, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:42:55.976634Z, time_modified: 2025-04-30T06:42:55.976634Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
21212025-04-30T06:42:57.104ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 372.567486ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"064d601f-0a53-448d-b924-c528a8e9e0e8","tls_certs":[]}},"default_endpoint":{"silo_id":"064d601f-0a53-448d-b924-c528a8e9e0e8","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo 064d601f-0a53-448d-b924-c528a8e9e0e8 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
21222025-04-30T06:42:57.104ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21232025-04-30T06:42:57.111ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21242025-04-30T06:42:57.111ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21252025-04-30T06:42:57.111ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21262025-04-30T06:42:57.112ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21272025-04-30T06:42:57.113ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b38b634-37ce-4dd1-a1a5-dabe389623ad", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21282025-04-30T06:42:57.113ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
21292025-04-30T06:42:57.133ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21302025-04-30T06:42:57.133ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21312025-04-30T06:42:57.134ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21322025-04-30T06:42:57.137ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21332025-04-30T06:42:57.137ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21342025-04-30T06:42:57.139ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21352025-04-30T06:42:57.142ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21362025-04-30T06:42:57.142ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21372025-04-30T06:42:57.143ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21382025-04-30T06:42:57.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22213f0d-66f2-4a2a-a51e-7ce1fd48154b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21392025-04-30T06:42:57.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
21402025-04-30T06:42:57.158ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21412025-04-30T06:42:57.158ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21422025-04-30T06:42:57.158ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21432025-04-30T06:42:57.158ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21442025-04-30T06:42:57.162ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21452025-04-30T06:42:57.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6956d9a-c923-42ef-9411-b7b883c6c041", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21462025-04-30T06:42:57.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
21472025-04-30T06:42:57.162ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21482025-04-30T06:42:57.162ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21492025-04-30T06:42:57.162ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21502025-04-30T06:42:57.164ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21512025-04-30T06:42:57.166ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21522025-04-30T06:42:57.166ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21532025-04-30T06:42:57.167ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21542025-04-30T06:42:57.167ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21552025-04-30T06:42:57.169ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a8d02af-0e13-4056-9433-79fa4e705f4d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21562025-04-30T06:42:57.169ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
21572025-04-30T06:42:57.171ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21582025-04-30T06:42:57.184ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21592025-04-30T06:42:57.184ZTRACtest_snapshot_expunge (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21602025-04-30T06:42:57.184ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21612025-04-30T06:42:57.184ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21622025-04-30T06:42:57.185ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21632025-04-30T06:42:57.188ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21642025-04-30T06:42:57.188ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21652025-04-30T06:42:57.189ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21662025-04-30T06:42:57.190ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f60430b-a5c0-4802-bbb0-46a08bc52aea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21672025-04-30T06:42:57.190ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
21682025-04-30T06:42:57.191ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21692025-04-30T06:42:57.191ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21702025-04-30T06:42:57.192ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21712025-04-30T06:42:57.197ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21722025-04-30T06:42:57.197ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21732025-04-30T06:42:57.199ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21742025-04-30T06:42:57.207ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06f25bd5-95dc-45aa-869a-6de5c6aba9f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21752025-04-30T06:42:57.207ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
21762025-04-30T06:42:57.208ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
21772025-04-30T06:42:57.208ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21782025-04-30T06:42:57.208ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21792025-04-30T06:42:57.208ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21802025-04-30T06:42:57.208ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21812025-04-30T06:42:57.214ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98615ba2-b705-471c-a7c6-8d94585d3ed9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21822025-04-30T06:42:57.214ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
21832025-04-30T06:42:57.215ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c355a21f-2c7c-47ed-a3ce-6ce2e773d95d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21842025-04-30T06:42:57.215ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
21852025-04-30T06:42:57.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b978b823-f13a-4ced-ba59-e84ac5cf1197", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21862025-04-30T06:42:57.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
21872025-04-30T06:42:57.218ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d543571-452a-4771-8a2d-98efcda1c399", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21882025-04-30T06:42:57.218ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
21892025-04-30T06:42:57.219ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21902025-04-30T06:42:57.219ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21912025-04-30T06:42:57.219ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21922025-04-30T06:42:57.221ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21932025-04-30T06:42:57.224ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d67634cd-57e5-4629-98bb-5e2697842bd2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
21942025-04-30T06:42:57.225ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
21952025-04-30T06:42:57.225ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21962025-04-30T06:42:57.225ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21972025-04-30T06:42:57.225ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21982025-04-30T06:42:57.226ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21992025-04-30T06:42:57.229ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22002025-04-30T06:42:57.240ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22012025-04-30T06:42:57.241ZTRACtest_snapshot_expunge (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22022025-04-30T06:42:57.241ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22032025-04-30T06:42:57.241ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22042025-04-30T06:42:57.241ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22052025-04-30T06:42:57.241ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22062025-04-30T06:42:57.241ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22072025-04-30T06:42:57.242ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a14444c-ddaa-48f1-b11f-dade13e0984c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22082025-04-30T06:42:57.242ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
22092025-04-30T06:42:57.245ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d080669d-f0c4-4140-85d2-5fd222591bdd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22102025-04-30T06:42:57.245ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
22112025-04-30T06:42:57.248ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e67e6f62-4f09-46d0-bda3-f6ba149f14a0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22122025-04-30T06:42:57.248ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
22132025-04-30T06:42:57.250ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c2e141d-256d-4758-b83f-386d3354aa7c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22142025-04-30T06:42:57.251ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
22152025-04-30T06:42:57.253ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22162025-04-30T06:42:57.253ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22172025-04-30T06:42:57.253ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22182025-04-30T06:42:57.255ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22192025-04-30T06:42:57.259ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7d6ae17-032e-421a-bf09-06dfc9179c8a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22202025-04-30T06:42:57.259ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
22212025-04-30T06:42:57.261ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc6f82f9-57de-4235-943a-4e66d7a62fdc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22222025-04-30T06:42:57.262ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
22232025-04-30T06:42:57.265ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2d09685-1ced-4d2f-a18e-c8be5704482a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22242025-04-30T06:42:57.266ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
22252025-04-30T06:42:57.266ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5166d38-4a4c-4922-b251-2f0e7e2ae672", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22262025-04-30T06:42:57.266ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
22272025-04-30T06:42:57.267ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e0f5e9-4b98-49a3-9b52-be89486756f7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22282025-04-30T06:42:57.268ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
22292025-04-30T06:42:57.269ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9e10b5a-1132-49d1-92b7-e818a0ce7261", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22302025-04-30T06:42:57.269ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
22312025-04-30T06:42:57.270ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61924b02-6c82-4476-aa39-fb99fd3af6de", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22322025-04-30T06:42:57.271ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
22332025-04-30T06:42:57.273ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cb2ffd9-08e4-49fb-939e-ddf0c7b1fce5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22342025-04-30T06:42:57.274ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
22352025-04-30T06:42:57.274ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22362025-04-30T06:42:57.275ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22372025-04-30T06:42:57.275ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22382025-04-30T06:42:57.275ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22392025-04-30T06:42:57.278ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22402025-04-30T06:42:57.287ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22412025-04-30T06:42:57.287ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22422025-04-30T06:42:57.287ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22432025-04-30T06:42:57.287ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22442025-04-30T06:42:57.288ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a12707d9-33e3-42da-92a8-d3f56261e04d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22452025-04-30T06:42:57.288ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
22462025-04-30T06:42:57.291ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b74f2f9-7516-45ee-9067-850196df6323", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22472025-04-30T06:42:57.291ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
22482025-04-30T06:42:57.293ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22492025-04-30T06:42:57.293ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22502025-04-30T06:42:57.293ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22512025-04-30T06:42:57.295ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "888a12ba-ba9d-4c95-9b18-03a37b45bbb0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22522025-04-30T06:42:57.296ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
22532025-04-30T06:42:57.296ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22542025-04-30T06:42:57.296ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22552025-04-30T06:42:57.296ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22562025-04-30T06:42:57.302ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6a0a6ca-3d58-4552-8389-08887c667f8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22572025-04-30T06:42:57.302ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
22582025-04-30T06:42:57.302ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22592025-04-30T06:42:57.302ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22602025-04-30T06:42:57.305ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22612025-04-30T06:42:57.308ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "759e71f4-0b85-4c6b-87f9-ccad50f89dc4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22622025-04-30T06:42:57.308ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
22632025-04-30T06:42:57.308ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22642025-04-30T06:42:57.308ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22652025-04-30T06:42:57.310ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22662025-04-30T06:42:57.314ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dab9a361-1ab2-43da-a570-513c38c700ff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22672025-04-30T06:42:57.314ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
22682025-04-30T06:42:57.315ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22692025-04-30T06:42:57.316ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22702025-04-30T06:42:57.316ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22712025-04-30T06:42:57.316ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22722025-04-30T06:42:57.317ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22732025-04-30T06:42:57.317ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4744a76-8726-4e9c-bbd0-032e50057325", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22742025-04-30T06:42:57.317ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
22752025-04-30T06:42:57.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02723279-77c9-4406-b074-e94a7bd6b464", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22762025-04-30T06:42:57.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
22772025-04-30T06:42:57.319ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22782025-04-30T06:42:57.319ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22792025-04-30T06:42:57.319ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22802025-04-30T06:42:57.320ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22812025-04-30T06:42:57.321ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72604f61-dd66-445d-9ebb-f69b2f0b0224", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22822025-04-30T06:42:57.321ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
22832025-04-30T06:42:57.322ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "547c8498-8ba6-421b-8cb8-75a8393918e3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22842025-04-30T06:42:57.322ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
22852025-04-30T06:42:57.322ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22862025-04-30T06:42:57.322ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22872025-04-30T06:42:57.322ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22882025-04-30T06:42:57.323ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22892025-04-30T06:42:57.326ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98f9af6f-30c1-478c-b9f3-93e2ab7e7216", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
22902025-04-30T06:42:57.326ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
22912025-04-30T06:42:57.326ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22922025-04-30T06:42:57.335ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22932025-04-30T06:42:57.335ZTRACtest_snapshot_expunge (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22942025-04-30T06:42:57.335ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22952025-04-30T06:42:57.335ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22962025-04-30T06:42:57.335ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22972025-04-30T06:42:57.336ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22982025-04-30T06:42:57.336ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22992025-04-30T06:42:57.339ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
23002025-04-30T06:42:57.341ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23012025-04-30T06:42:57.341ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23022025-04-30T06:42:57.342ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23032025-04-30T06:42:57.344ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23042025-04-30T06:42:57.344ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23052025-04-30T06:42:57.344ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23062025-04-30T06:42:57.345ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23072025-04-30T06:42:57.345ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ddf6b1c-54b8-49b9-bb9f-bf734576fc5c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23082025-04-30T06:42:57.345ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
23092025-04-30T06:42:57.349ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e1c3500-b590-429c-95f5-4c44f99d4183", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23102025-04-30T06:42:57.349ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
23112025-04-30T06:42:57.349ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23122025-04-30T06:42:57.364ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
23132025-04-30T06:42:57.364ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23142025-04-30T06:42:57.364ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23152025-04-30T06:42:57.365ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23162025-04-30T06:42:57.368ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6749ab42-5fb7-408a-99cc-8663d09c773e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23172025-04-30T06:42:57.369ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
23182025-04-30T06:42:57.375ZDEBGtest_snapshot_expunge (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 736df47b-f45d-4c92-840a-837099043a36, kind: Service { id: 58f2365f-b720-4ec1-ad50-a68fadcacb1c }, name: Name("nexus-58f2365f-b720-4ec1-ad50-a68fadcacb1c"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: b93f313f-d062-4189-939b-bc4fddea0705, kind: Service { id: 69aed484-7768-456c-9a68-bca265ad4317 }, name: Name("external-dns-69aed484-7768-456c-9a68-bca265ad4317"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
23192025-04-30T06:42:57.375ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
23202025-04-30T06:42:57.375ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23212025-04-30T06:42:57.376ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23222025-04-30T06:42:57.376ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23232025-04-30T06:42:57.376ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23242025-04-30T06:42:57.377ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3d9a12f-05e7-46cf-ae8c-0cf21b069be7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23252025-04-30T06:42:57.377ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
23262025-04-30T06:42:57.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed51b660-a8ea-4c1e-853d-6992d0a069c6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23272025-04-30T06:42:57.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
23282025-04-30T06:42:57.383ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23292025-04-30T06:42:57.388ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23302025-04-30T06:42:57.388ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23312025-04-30T06:42:57.389ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23322025-04-30T06:42:57.389ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23332025-04-30T06:42:57.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a35de3e-56bb-4cea-901d-7fb471ed69ea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23342025-04-30T06:42:57.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
23352025-04-30T06:42:57.391ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "addb5dd8-c2c7-48ae-808f-c9a4e1b1c0a9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23362025-04-30T06:42:57.391ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
23372025-04-30T06:42:57.395ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ad07f90-d197-448a-ac1d-5a201df29545", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23382025-04-30T06:42:57.395ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
23392025-04-30T06:42:57.395ZTRACtest_snapshot_expunge (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:42:56.241733Z, allowed_ips: None }
23402025-04-30T06:42:57.395ZDEBGtest_snapshot_expunge (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23412025-04-30T06:42:57.395ZDEBGtest_snapshot_expunge (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: b93f313f-d062-4189-939b-bc4fddea0705, kind: Service { id: 69aed484-7768-456c-9a68-bca265ad4317 }, name: Name("external-dns-69aed484-7768-456c-9a68-bca265ad4317"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 736df47b-f45d-4c92-840a-837099043a36, kind: Service { id: 58f2365f-b720-4ec1-ad50-a68fadcacb1c }, name: Name("nexus-58f2365f-b720-4ec1-ad50-a68fadcacb1c"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23422025-04-30T06:42:57.395ZDEBGtest_snapshot_expunge (InternalApi): resolved 2 rules for sleds
23432025-04-30T06:42:57.397ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e603d8f-8400-4550-be49-418216c9d4bc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23442025-04-30T06:42:57.397ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
23452025-04-30T06:42:57.399ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f1e3ec4-e65e-4c40-a0cf-4885e148b63d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23462025-04-30T06:42:57.399ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
23472025-04-30T06:42:57.399ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23482025-04-30T06:42:57.399ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23492025-04-30T06:42:57.399ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23502025-04-30T06:42:57.402ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23512025-04-30T06:42:57.410ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23522025-04-30T06:42:57.419ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
23532025-04-30T06:42:57.419ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
23542025-04-30T06:42:57.420ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23552025-04-30T06:42:57.420ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23562025-04-30T06:42:57.420ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23572025-04-30T06:42:57.420ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23582025-04-30T06:42:57.423ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23592025-04-30T06:42:57.428ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
23602025-04-30T06:42:57.428ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23612025-04-30T06:42:57.428ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23622025-04-30T06:42:57.429ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23632025-04-30T06:42:57.510ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
23642025-04-30T06:42:57.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
23652025-04-30T06:42:57.510ZDEBGtest_snapshot_expunge (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:42:49.972333Z, time_modified: 2025-04-30T06:42:49.972333Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(51119), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(42164) }]
23662025-04-30T06:42:57.510ZDEBGtest_snapshot_expunge (InternalApi): sending firewall rules to sled agents
23672025-04-30T06:42:57.510ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23682025-04-30T06:42:57.510ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23692025-04-30T06:42:57.512ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23702025-04-30T06:42:57.512ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:60866 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23712025-04-30T06:42:57.513ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:60866 req_id = 741caeab-f821-4bad-b488-885d31951e64 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
23722025-04-30T06:42:57.513ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 439 local_addr = [::1]:51119 method = GET remote_addr = [::1]:60866 req_id = 741caeab-f821-4bad-b488-885d31951e64 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
23732025-04-30T06:42:57.513ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "741caeab-f821-4bad-b488-885d31951e64", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23742025-04-30T06:42:57.516ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
23752025-04-30T06:42:57.516ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
23762025-04-30T06:42:57.516ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
23772025-04-30T06:42:57.519ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
23782025-04-30T06:42:57.519ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23792025-04-30T06:42:57.519ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23802025-04-30T06:42:57.519ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23812025-04-30T06:42:57.519ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23822025-04-30T06:42:57.524ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23832025-04-30T06:42:57.529ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
23842025-04-30T06:42:57.530ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23852025-04-30T06:42:57.530ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23862025-04-30T06:42:57.531ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23872025-04-30T06:42:57.531ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23882025-04-30T06:42:57.531ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23892025-04-30T06:42:57.531ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23902025-04-30T06:42:57.532ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23912025-04-30T06:42:57.532ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23922025-04-30T06:42:57.535ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23932025-04-30T06:42:57.540ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23942025-04-30T06:42:57.599ZDEBGtest_snapshot_expunge (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:51119/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23952025-04-30T06:42:57.601ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:59295 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23962025-04-30T06:42:57.601ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = PUT remote_addr = [::1]:59295 req_id = cd86eb22-203a-4f05-80b2-3705b304b48c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23972025-04-30T06:42:57.602ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 801 local_addr = [::1]:51119 method = PUT remote_addr = [::1]:59295 req_id = cd86eb22-203a-4f05-80b2-3705b304b48c response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23982025-04-30T06:42:57.602ZDEBGtest_snapshot_expunge (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:51119/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "cd86eb22-203a-4f05-80b2-3705b304b48c", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
23992025-04-30T06:42:57.603ZINFOtest_snapshot_expunge (InternalApi): updated firewall rules on 1 sleds
24002025-04-30T06:42:57.603ZINFOtest_snapshot_expunge (nexus): plumbed initial IP allowlist
24012025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = / versions = all
24022025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /access versions = all
24032025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /assets/{path} versions = all
24042025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /device/auth versions = all
24052025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /device/confirm versions = all
24062025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /device/success versions = all
24072025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /device/token versions = all
24082025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /device/verify versions = all
24092025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /experimental/v1/probes versions = all
24102025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /experimental/v1/probes versions = all
24112025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /experimental/v1/probes/{probe} versions = all
24122025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /experimental/v1/probes/{probe} versions = all
24132025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /experimental/v1/system/support-bundles versions = all
24142025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /experimental/v1/system/support-bundles versions = all
24152025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
24162025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
24172025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
24182025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
24192025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
24202025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
24212025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
24222025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /images versions = all
24232025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /login versions = all
24242025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /login/{silo_name}/local versions = all
24252025-04-30T06:42:57.671ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
24262025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
24272025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
24282025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /lookup/{path} versions = all
24292025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /projects/{path} versions = all
24302025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /projects-new versions = all
24312025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /settings/{path} versions = all
24322025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /system/{path} versions = all
24332025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /utilization versions = all
24342025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/affinity-groups versions = all
24352025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/affinity-groups versions = all
24362025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
24372025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
24382025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
24392025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
24402025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24412025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24422025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24432025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/anti-affinity-groups versions = all
24442025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/anti-affinity-groups versions = all
24452025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24462025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24472025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24482025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24492025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24502025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24512025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24522025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/certificates versions = all
24532025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/certificates versions = all
24542025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/certificates/{certificate} versions = all
24552025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/certificates/{certificate} versions = all
24562025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/disks versions = all
24572025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/disks versions = all
24582025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/disks/{disk} versions = all
24592025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/disks/{disk} versions = all
24602025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24612025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24622025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24632025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/disks/{disk}/finalize versions = all
24642025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24652025-04-30T06:42:57.672ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/floating-ips versions = all
24662025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/floating-ips versions = all
24672025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24682025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24692025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24702025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24712025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24722025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/groups versions = all
24732025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/groups/{group_id} versions = all
24742025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/images versions = all
24752025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/images versions = all
24762025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/images/{image} versions = all
24772025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/images/{image} versions = all
24782025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/images/{image}/demote versions = all
24792025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/images/{image}/promote versions = all
24802025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/instances versions = all
24812025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/instances versions = all
24822025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/instances/{instance} versions = all
24832025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/instances/{instance} versions = all
24842025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/instances/{instance} versions = all
24852025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24862025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24872025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/instances/{instance}/disks versions = all
24882025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24892025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24902025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/instances/{instance}/external-ips versions = all
24912025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24922025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24932025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/instances/{instance}/reboot versions = all
24942025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/instances/{instance}/serial-console versions = all
24952025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24962025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24972025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/instances/{instance}/start versions = all
24982025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/instances/{instance}/stop versions = all
24992025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/internet-gateway-ip-addresses versions = all
25002025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/internet-gateway-ip-addresses versions = all
25012025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
25022025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/internet-gateway-ip-pools versions = all
25032025-04-30T06:42:57.673ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/internet-gateway-ip-pools versions = all
25042025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
25052025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/internet-gateways versions = all
25062025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/internet-gateways versions = all
25072025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
25082025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/internet-gateways/{gateway} versions = all
25092025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/ip-pools versions = all
25102025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/ip-pools/{pool} versions = all
25112025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/login/{silo_name}/local versions = all
25122025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/logout versions = all
25132025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/me versions = all
25142025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/me/groups versions = all
25152025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/me/ssh-keys versions = all
25162025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/me/ssh-keys versions = all
25172025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
25182025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
25192025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/metrics/{metric_name} versions = all
25202025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/network-interfaces versions = all
25212025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/network-interfaces versions = all
25222025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/network-interfaces/{interface} versions = all
25232025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/network-interfaces/{interface} versions = all
25242025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/network-interfaces/{interface} versions = all
25252025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/ping versions = all
25262025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/policy versions = all
25272025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/policy versions = all
25282025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/projects versions = all
25292025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/projects versions = all
25302025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/projects/{project} versions = all
25312025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/projects/{project} versions = all
25322025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/projects/{project} versions = all
25332025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/projects/{project}/policy versions = all
25342025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/projects/{project}/policy versions = all
25352025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/snapshots versions = all
25362025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/snapshots versions = all
25372025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/snapshots/{snapshot} versions = all
25382025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/snapshots/{snapshot} versions = all
25392025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/disks versions = all
25402025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
25412025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
25422025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/racks versions = all
25432025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
25442025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/sleds versions = all
25452025-04-30T06:42:57.674ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/hardware/sleds versions = all
25462025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
25472025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
25482025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25492025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25502025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25512025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/switch-port versions = all
25522025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25532025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25542025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25552025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25562025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25572025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/switches versions = all
25582025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25592025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/identity-providers versions = all
25602025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/identity-providers/local/users versions = all
25612025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25622025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25632025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/identity-providers/saml versions = all
25642025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25652025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/ip-pools versions = all
25662025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/ip-pools versions = all
25672025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25682025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/ip-pools/{pool} versions = all
25692025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25702025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25712025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25722025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25732025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25742025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25752025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25762025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25772025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25782025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/ip-pools-service versions = all
25792025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25802025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25812025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25822025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/metrics/{metric_name} versions = all
25832025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/address-lot versions = all
25842025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/networking/address-lot versions = all
25852025-04-30T06:42:57.675ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25862025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25872025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/allow-list versions = all
25882025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/networking/allow-list versions = all
25892025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/networking/bfd-disable versions = all
25902025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/networking/bfd-enable versions = all
25912025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/bfd-status versions = all
25922025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/networking/bgp versions = all
25932025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/bgp versions = all
25942025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/networking/bgp versions = all
25952025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25962025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25972025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25982025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25992025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/bgp-exported versions = all
26002025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/bgp-message-history versions = all
26012025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
26022025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/bgp-status versions = all
26032025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/loopback-address versions = all
26042025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/networking/loopback-address versions = all
26052025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
26062025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
26072025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/switch-port-settings versions = all
26082025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/networking/switch-port-settings versions = all
26092025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
26102025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/policy versions = all
26112025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/policy versions = all
26122025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/roles versions = all
26132025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/roles/{role_name} versions = all
26142025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/silo-quotas versions = all
26152025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/silos versions = all
26162025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/silos versions = all
26172025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/system/silos/{silo} versions = all
26182025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/silos/{silo} versions = all
26192025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
26202025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/silos/{silo}/policy versions = all
26212025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/silos/{silo}/policy versions = all
26222025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/silos/{silo}/quotas versions = all
26232025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
26242025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/system/timeseries/query versions = all
26252025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/timeseries/schemas versions = all
26262025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/update/repository versions = all
26272025-04-30T06:42:57.676ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/update/repository/{system_version} versions = all
26282025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/update/target-release versions = all
26292025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/system/update/target-release versions = all
26302025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/users versions = all
26312025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/users/{user_id} versions = all
26322025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/users-builtin versions = all
26332025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/users-builtin/{user} versions = all
26342025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/utilization/silos versions = all
26352025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/system/utilization/silos/{silo} versions = all
26362025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/timeseries/query versions = all
26372025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/users versions = all
26382025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/utilization versions = all
26392025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpc-firewall-rules versions = all
26402025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/vpc-firewall-rules versions = all
26412025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpc-router-routes versions = all
26422025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/vpc-router-routes versions = all
26432025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
26442025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpc-router-routes/{route} versions = all
26452025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/vpc-router-routes/{route} versions = all
26462025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpc-routers versions = all
26472025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/vpc-routers versions = all
26482025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/vpc-routers/{router} versions = all
26492025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpc-routers/{router} versions = all
26502025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/vpc-routers/{router} versions = all
26512025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpc-subnets versions = all
26522025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/vpc-subnets versions = all
26532025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26542025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26552025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26562025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26572025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpcs versions = all
26582025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/vpcs versions = all
26592025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/vpcs/{vpc} versions = all
26602025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/vpcs/{vpc} versions = all
26612025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/vpcs/{vpc} versions = all
26622025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/webhooks/deliveries versions = all
26632025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26642025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/webhooks/event-classes versions = all
26652025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/webhooks/receivers versions = all
26662025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/webhooks/receivers versions = all
26672025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26682025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26692025-04-30T06:42:57.677ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26702025-04-30T06:42:57.678ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26712025-04-30T06:42:57.678ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26722025-04-30T06:42:57.678ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26732025-04-30T06:42:57.678ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = GET path = /v1/webhooks/secrets versions = all
26742025-04-30T06:42:57.678ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = POST path = /v1/webhooks/secrets versions = all
26752025-04-30T06:42:57.678ZDEBGtest_snapshot_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:37576 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26762025-04-30T06:42:57.678ZINFOtest_snapshot_expunge (dropshot_external): listening local_addr = 127.0.0.1:37576
26772025-04-30T06:42:57.678ZDEBGtest_snapshot_expunge (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:37576
26782025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = / versions = all
26792025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /access versions = all
26802025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /assets/{path} versions = all
26812025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /device/auth versions = all
26822025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /device/confirm versions = all
26832025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /device/success versions = all
26842025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /device/token versions = all
26852025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /device/verify versions = all
26862025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /experimental/v1/probes versions = all
26872025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /experimental/v1/probes versions = all
26882025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26892025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /experimental/v1/probes/{probe} versions = all
26902025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /experimental/v1/system/support-bundles versions = all
26912025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /experimental/v1/system/support-bundles versions = all
26922025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26932025-04-30T06:42:57.763ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26942025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26952025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26962025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26972025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26982025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26992025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /images versions = all
27002025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /login versions = all
27012025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /login/{silo_name}/local versions = all
27022025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
27032025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
27042025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
27052025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /lookup/{path} versions = all
27062025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /projects/{path} versions = all
27072025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /projects-new versions = all
27082025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /settings/{path} versions = all
27092025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /system/{path} versions = all
27102025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /utilization versions = all
27112025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/affinity-groups versions = all
27122025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/affinity-groups versions = all
27132025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
27142025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
27152025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
27162025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
27172025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27182025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27192025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27202025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/anti-affinity-groups versions = all
27212025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/anti-affinity-groups versions = all
27222025-04-30T06:42:57.764ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27232025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27242025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27252025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
27262025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27272025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27282025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27292025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/certificates versions = all
27302025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/certificates versions = all
27312025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/certificates/{certificate} versions = all
27322025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/certificates/{certificate} versions = all
27332025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/disks versions = all
27342025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/disks versions = all
27352025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/disks/{disk} versions = all
27362025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/disks/{disk} versions = all
27372025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/disks/{disk}/bulk-write versions = all
27382025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
27392025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
27402025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/disks/{disk}/finalize versions = all
27412025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
27422025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/floating-ips versions = all
27432025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/floating-ips versions = all
27442025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
27452025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/floating-ips/{floating_ip} versions = all
27462025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
27472025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
27482025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27492025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/groups versions = all
27502025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/groups/{group_id} versions = all
27512025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/images versions = all
27522025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/images versions = all
27532025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/images/{image} versions = all
27542025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/images/{image} versions = all
27552025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/images/{image}/demote versions = all
27562025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/images/{image}/promote versions = all
27572025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/instances versions = all
27582025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/instances versions = all
27592025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/instances/{instance} versions = all
27602025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/instances/{instance} versions = all
27612025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/instances/{instance} versions = all
27622025-04-30T06:42:57.765ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27632025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27642025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/instances/{instance}/disks versions = all
27652025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27662025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27672025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/instances/{instance}/external-ips versions = all
27682025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27692025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27702025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/instances/{instance}/reboot versions = all
27712025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/instances/{instance}/serial-console versions = all
27722025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27732025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27742025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/instances/{instance}/start versions = all
27752025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/instances/{instance}/stop versions = all
27762025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27772025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27782025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27792025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/internet-gateway-ip-pools versions = all
27802025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/internet-gateway-ip-pools versions = all
27812025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27822025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/internet-gateways versions = all
27832025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/internet-gateways versions = all
27842025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27852025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/internet-gateways/{gateway} versions = all
27862025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/ip-pools versions = all
27872025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/ip-pools/{pool} versions = all
27882025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/login/{silo_name}/local versions = all
27892025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/logout versions = all
27902025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/me versions = all
27912025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/me/groups versions = all
27922025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/me/ssh-keys versions = all
27932025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/me/ssh-keys versions = all
27942025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27952025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27962025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/metrics/{metric_name} versions = all
27972025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/network-interfaces versions = all
27982025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/network-interfaces versions = all
27992025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/network-interfaces/{interface} versions = all
28002025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/network-interfaces/{interface} versions = all
28012025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/network-interfaces/{interface} versions = all
28022025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/ping versions = all
28032025-04-30T06:42:57.766ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/policy versions = all
28042025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/policy versions = all
28052025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/projects versions = all
28062025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/projects versions = all
28072025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/projects/{project} versions = all
28082025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/projects/{project} versions = all
28092025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/projects/{project} versions = all
28102025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/projects/{project}/policy versions = all
28112025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/projects/{project}/policy versions = all
28122025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/snapshots versions = all
28132025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/snapshots versions = all
28142025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/snapshots/{snapshot} versions = all
28152025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/snapshots/{snapshot} versions = all
28162025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/disks versions = all
28172025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
28182025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
28192025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/racks versions = all
28202025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
28212025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/sleds versions = all
28222025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/hardware/sleds versions = all
28232025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
28242025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
28252025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
28262025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
28272025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
28282025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/switch-port versions = all
28292025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
28302025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
28312025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
28322025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
28332025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
28342025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/switches versions = all
28352025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
28362025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/identity-providers versions = all
28372025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/identity-providers/local/users versions = all
28382025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
28392025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
28402025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/identity-providers/saml versions = all
28412025-04-30T06:42:57.767ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
28422025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/ip-pools versions = all
28432025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/ip-pools versions = all
28442025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
28452025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/ip-pools/{pool} versions = all
28462025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/ip-pools/{pool} versions = all
28472025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
28482025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28492025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28502025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28512025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28522025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28532025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28542025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28552025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/ip-pools-service versions = all
28562025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28572025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28582025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28592025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/metrics/{metric_name} versions = all
28602025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/address-lot versions = all
28612025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/networking/address-lot versions = all
28622025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28632025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28642025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/allow-list versions = all
28652025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/networking/allow-list versions = all
28662025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/networking/bfd-disable versions = all
28672025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/networking/bfd-enable versions = all
28682025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/bfd-status versions = all
28692025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/networking/bgp versions = all
28702025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/bgp versions = all
28712025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/networking/bgp versions = all
28722025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28732025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28742025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28752025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28762025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/bgp-exported versions = all
28772025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/bgp-message-history versions = all
28782025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28792025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/bgp-status versions = all
28802025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/loopback-address versions = all
28812025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/networking/loopback-address versions = all
28822025-04-30T06:42:57.768ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28832025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28842025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/switch-port-settings versions = all
28852025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/networking/switch-port-settings versions = all
28862025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28872025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/policy versions = all
28882025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/policy versions = all
28892025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/roles versions = all
28902025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/roles/{role_name} versions = all
28912025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/silo-quotas versions = all
28922025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/silos versions = all
28932025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/silos versions = all
28942025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/system/silos/{silo} versions = all
28952025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/silos/{silo} versions = all
28962025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28972025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/silos/{silo}/policy versions = all
28982025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28992025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/silos/{silo}/quotas versions = all
29002025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
29012025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/system/timeseries/query versions = all
29022025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/timeseries/schemas versions = all
29032025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/update/repository versions = all
29042025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/update/repository/{system_version} versions = all
29052025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/update/target-release versions = all
29062025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/system/update/target-release versions = all
29072025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/users versions = all
29082025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/users/{user_id} versions = all
29092025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/users-builtin versions = all
29102025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/users-builtin/{user} versions = all
29112025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/utilization/silos versions = all
29122025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/system/utilization/silos/{silo} versions = all
29132025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/timeseries/query versions = all
29142025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/users versions = all
29152025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/utilization versions = all
29162025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpc-firewall-rules versions = all
29172025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/vpc-firewall-rules versions = all
29182025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpc-router-routes versions = all
29192025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/vpc-router-routes versions = all
29202025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
29212025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpc-router-routes/{route} versions = all
29222025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/vpc-router-routes/{route} versions = all
29232025-04-30T06:42:57.769ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpc-routers versions = all
29242025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/vpc-routers versions = all
29252025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/vpc-routers/{router} versions = all
29262025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpc-routers/{router} versions = all
29272025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/vpc-routers/{router} versions = all
29282025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpc-subnets versions = all
29292025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/vpc-subnets versions = all
29302025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
29312025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpc-subnets/{subnet} versions = all
29322025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
29332025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
29342025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpcs versions = all
29352025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/vpcs versions = all
29362025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/vpcs/{vpc} versions = all
29372025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/vpcs/{vpc} versions = all
29382025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/vpcs/{vpc} versions = all
29392025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/webhooks/deliveries versions = all
29402025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
29412025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/webhooks/event-classes versions = all
29422025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/webhooks/receivers versions = all
29432025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/webhooks/receivers versions = all
29442025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
29452025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
29462025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
29472025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
29482025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29492025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29502025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = GET path = /v1/webhooks/secrets versions = all
29512025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = POST path = /v1/webhooks/secrets versions = all
29522025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:56547 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29532025-04-30T06:42:57.770ZINFOtest_snapshot_expunge (dropshot_external_techport): listening local_addr = 127.0.0.1:56547
29542025-04-30T06:42:57.770ZDEBGtest_snapshot_expunge (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:56547
29552025-04-30T06:42:57.772ZDEBGtest_snapshot_expunge (nexus-producer-server): registered DTrace probes
29562025-04-30T06:42:57.772ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = 127.0.0.1:47175 method = GET path = /{producer_id} versions = all
29572025-04-30T06:42:57.772ZINFOtest_snapshot_expunge (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:47175
29582025-04-30T06:42:57.774ZDEBGtest_snapshot_expunge (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:47175
29592025-04-30T06:42:57.774ZDEBGtest_snapshot_expunge (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:47175
29602025-04-30T06:42:57.774ZDEBGtest_snapshot_expunge (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:41836
29612025-04-30T06:42:57.774ZDEBGtest_snapshot_expunge (nexus-producer-server): starting producer registration task
29622025-04-30T06:42:57.774ZINFOtest_snapshot_expunge (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:47175 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
29632025-04-30T06:42:57.774ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_oximeter
29642025-04-30T06:42:57.774ZDEBGtest_snapshot_expunge: Starting Oximeter
29652025-04-30T06:42:57.775ZDEBGtest_snapshot_expunge (oximeter): registered DTrace probes
29662025-04-30T06:42:57.775ZINFOtest_snapshot_expunge (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29672025-04-30T06:42:57.775ZDEBGtest_snapshot_expunge (oximeter): creating ClickHouse client
29682025-04-30T06:42:57.776ZDEBGtest_snapshot_expunge (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
29692025-04-30T06:42:57.777ZDEBGtest_snapshot_expunge (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29702025-04-30T06:42:57.777ZDEBGtest_snapshot_expunge (producer-registration-task): will register with Nexus at 127.0.0.1:41836
29712025-04-30T06:42:57.837ZDEBGtest_snapshot_expunge (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:41836/metrics/producers
29722025-04-30T06:42:57.838ZINFOtest_snapshot_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:41836 remote_addr = 127.0.0.1:63037
29732025-04-30T06:42:57.840ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:63037 req_id = 77209dfd-fc56-47a4-8bbc-2b1066ac7f5e uri = /metrics/producers
29742025-04-30T06:42:57.841ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:63037 req_id = 77209dfd-fc56-47a4-8bbc-2b1066ac7f5e resource = Database uri = /metrics/producers
29752025-04-30T06:42:57.841ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:63037 req_id = 77209dfd-fc56-47a4-8bbc-2b1066ac7f5e roles = RoleSet { roles: {} } uri = /metrics/producers
29762025-04-30T06:42:57.842ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:63037 req_id = 77209dfd-fc56-47a4-8bbc-2b1066ac7f5e resource = Database result = Ok(()) uri = /metrics/producers
29772025-04-30T06:42:57.846ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29782025-04-30T06:42:57.860ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: ce d4 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:52313 req_id = 363d8281-99b7-44c7-987c-3e23547afee4
29792025-04-30T06:42:57.860ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52948,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52313 req_id = 363d8281-99b7-44c7-987c-3e23547afee4
29802025-04-30T06:42:57.861ZDEBGtest_snapshot_expunge (store): query key key = _crucible-pantry._tcp
29812025-04-30T06:42:57.861ZDEBGtest_snapshot_expunge (store): query key key = fd382bc8-252a-498d-a152-387b7e5fb18a.host
29822025-04-30T06:42:57.861ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("fd382bc8-252a-498d-a152-387b7e5fb18a.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52313 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 54156, target: Name("fd382bc8-252a-498d-a152-387b7e5fb18a.host.control-plane.oxide.internal") })) }] req_id = 363d8281-99b7-44c7-987c-3e23547afee4
29832025-04-30T06:42:57.861ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 6e aa 01 00 00 01 00 00 00 00 00 01 24 66 64 33 n...........$fd3\n0010: 38 32 62 63 38 2d 32 35 32 61 2d 34 39 38 64 2d 82bc8-252a-498d-\n0020: 61 31 35 32 2d 33 38 37 62 37 65 35 66 62 31 38 a152-387b7e5fb18\n0030: 61 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 a.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:50650 req_id = 98afc477-b434-443c-982c-36fc1133e632
29842025-04-30T06:42:57.861ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28330,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("fd382bc8-252a-498d-a152-387b7e5fb18a.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("fd382bc8-252a-498d-a152-387b7e5fb18a.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 102,\n 100,\n 51,\n 56,\n 50,\n 98,\n 99,\n 56,\n 45,\n 50,\n 53,\n 50,\n 97,\n 45,\n 52,\n 57,\n 56,\n 100,\n 45,\n 97,\n 49,\n 53,\n 50,\n 45,\n 51,\n 56,\n 55,\n 98,\n 55,\n 101,\n 53,\n 102,\n 98,\n 49,\n 56,\n 97,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50650 req_id = 98afc477-b434-443c-982c-36fc1133e632
29852025-04-30T06:42:57.861ZDEBGtest_snapshot_expunge (store): query key key = fd382bc8-252a-498d-a152-387b7e5fb18a.host
29862025-04-30T06:42:57.861ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50650 query = LowerQuery { name: LowerName(Name("fd382bc8-252a-498d-a152-387b7e5fb18a.host.control-plane.oxide.internal.")), original: Query { name: Name("fd382bc8-252a-498d-a152-387b7e5fb18a.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("fd382bc8-252a-498d-a152-387b7e5fb18a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 98afc477-b434-443c-982c-36fc1133e632
29872025-04-30T06:42:57.942ZINFOtest_snapshot_expunge (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 102145 local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:63037 req_id = 77209dfd-fc56-47a4-8bbc-2b1066ac7f5e response_code = 503 uri = /metrics/producers
29882025-04-30T06:42:57.943ZWARNtest_snapshot_expunge (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = f2d4d5c2-3547-4831-8529-796764280fb6
29892025-04-30T06:42:57.944ZDEBGtest_snapshot_expunge (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29902025-04-30T06:42:57.944ZINFOtest_snapshot_expunge (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = f2d4d5c2-3547-4831-8529-796764280fb6
29912025-04-30T06:42:57.944ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:35992 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29922025-04-30T06:42:57.944ZDEBGtest_snapshot_expunge (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "77209dfd-fc56-47a4-8bbc-2b1066ac7f5e", "content-length": "133", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
29932025-04-30T06:42:57.946ZWARNtest_snapshot_expunge (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 169.067896ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"77209dfd-fc56-47a4-8bbc-2b1066ac7f5e\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:42:57 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"77209dfd-fc56-47a4-8bbc-2b1066ac7f5e\\" }" file = oximeter/producer/src/lib.rs:371
29942025-04-30T06:42:57.946ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29952025-04-30T06:42:57.947ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:35992 req_id = ee641bc5-20b1-493e-a653-7532d5eb754f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29962025-04-30T06:42:57.947ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 419 local_addr = [::1]:54156 method = GET remote_addr = [::1]:35992 req_id = ee641bc5-20b1-493e-a653-7532d5eb754f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
29972025-04-30T06:42:58.340ZWARNtest_snapshot_expunge (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = f2d4d5c2-3547-4831-8529-796764280fb6
29982025-04-30T06:42:58.340ZINFOtest_snapshot_expunge (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = f2d4d5c2-3547-4831-8529-796764280fb6 version = 0
29992025-04-30T06:42:58.340ZINFOtest_snapshot_expunge (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = f2d4d5c2-3547-4831-8529-796764280fb6
30002025-04-30T06:42:58.340ZDEBGtest_snapshot_expunge (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
30012025-04-30T06:42:58.341ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:63006 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30022025-04-30T06:42:58.341ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:42741 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30032025-04-30T06:42:58.341ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:53414 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30042025-04-30T06:42:58.341ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:39483 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30052025-04-30T06:42:58.341ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:42266 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30062025-04-30T06:42:58.341ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:33318 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30072025-04-30T06:42:58.341ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:48132 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30082025-04-30T06:42:58.341ZDEBGtest_snapshot_expunge (producer-registration-task): will register with Nexus at 127.0.0.1:41836
30092025-04-30T06:42:58.391ZDEBGtest_snapshot_expunge (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:41836/metrics/producers
30102025-04-30T06:42:58.393ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = CREATE DATABASE IF NOT EXISTS oximeter
30112025-04-30T06:42:58.394ZINFOtest_snapshot_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:41836 remote_addr = 127.0.0.1:49614
30122025-04-30T06:42:58.397ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:63006 req_id = 5f6b38f2-18d9-4a77-807e-ee3617cd2001 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30132025-04-30T06:42:58.398ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:42741 req_id = fe4f7b0d-750d-4eeb-8d33-2a71c00d8893 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30142025-04-30T06:42:58.402ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:53414 req_id = 766a6b73-9a43-4e45-8f1f-d35125c1ccd7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30152025-04-30T06:42:58.402ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:39483 req_id = 4b8aed04-6821-4e97-a02b-6da2b96bcb0a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30162025-04-30T06:42:58.402ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:42266 req_id = d0abb9f4-7782-4e10-8087-b26d68c25102 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30172025-04-30T06:42:58.402ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:33318 req_id = cde1c2a0-e7ce-4d7b-a868-002a21441d5c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30182025-04-30T06:42:58.402ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:48132 req_id = f2b61441-293e-4acc-b4f1-b753a1ce7686 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30192025-04-30T06:42:58.402ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 4805 local_addr = [::1]:54156 method = GET remote_addr = [::1]:63006 req_id = 5f6b38f2-18d9-4a77-807e-ee3617cd2001 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30202025-04-30T06:42:58.402ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 4325 local_addr = [::1]:54156 method = GET remote_addr = [::1]:42741 req_id = fe4f7b0d-750d-4eeb-8d33-2a71c00d8893 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30212025-04-30T06:42:58.402ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 992 local_addr = [::1]:54156 method = GET remote_addr = [::1]:53414 req_id = 766a6b73-9a43-4e45-8f1f-d35125c1ccd7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30222025-04-30T06:42:58.402ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 959 local_addr = [::1]:54156 method = GET remote_addr = [::1]:39483 req_id = 4b8aed04-6821-4e97-a02b-6da2b96bcb0a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30232025-04-30T06:42:58.402ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 819 local_addr = [::1]:54156 method = GET remote_addr = [::1]:42266 req_id = d0abb9f4-7782-4e10-8087-b26d68c25102 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30242025-04-30T06:42:58.402ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 755 local_addr = [::1]:54156 method = GET remote_addr = [::1]:33318 req_id = cde1c2a0-e7ce-4d7b-a868-002a21441d5c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30252025-04-30T06:42:58.402ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 789 local_addr = [::1]:54156 method = GET remote_addr = [::1]:48132 req_id = f2b61441-293e-4acc-b4f1-b753a1ce7686 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30262025-04-30T06:42:58.403ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
30272025-04-30T06:42:58.409ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:49614 req_id = e012f25d-9a5f-41d0-b7e4-09c249920434 uri = /metrics/producers
30282025-04-30T06:42:58.409ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:49614 req_id = e012f25d-9a5f-41d0-b7e4-09c249920434 resource = Database uri = /metrics/producers
30292025-04-30T06:42:58.409ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:49614 req_id = e012f25d-9a5f-41d0-b7e4-09c249920434 roles = RoleSet { roles: {} } uri = /metrics/producers
30302025-04-30T06:42:58.410ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:49614 req_id = e012f25d-9a5f-41d0-b7e4-09c249920434 resource = Database result = Ok(()) uri = /metrics/producers
30312025-04-30T06:42:58.411ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30322025-04-30T06:42:58.428ZINFOtest_snapshot_expunge (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 22378 local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:49614 req_id = e012f25d-9a5f-41d0-b7e4-09c249920434 response_code = 503 uri = /metrics/producers
30332025-04-30T06:42:58.430ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30342025-04-30T06:42:58.435ZDEBGtest_snapshot_expunge (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "e012f25d-9a5f-41d0-b7e4-09c249920434", "content-length": "133", "date": "Wed, 30 Apr 2025 06:42:57 GMT"} })
30352025-04-30T06:42:58.437ZWARNtest_snapshot_expunge (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 660.54877ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"e012f25d-9a5f-41d0-b7e4-09c249920434\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:42:57 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"e012f25d-9a5f-41d0-b7e4-09c249920434\\" }" file = oximeter/producer/src/lib.rs:371
30362025-04-30T06:42:58.437ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30372025-04-30T06:42:58.452ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30382025-04-30T06:42:58.459ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30392025-04-30T06:42:58.464ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30402025-04-30T06:42:58.473ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30412025-04-30T06:42:58.474ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
30422025-04-30T06:42:58.475ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
30432025-04-30T06:42:58.475ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.742324537s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
30442025-04-30T06:42:58.479ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30452025-04-30T06:42:58.487ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30462025-04-30T06:42:58.491ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30472025-04-30T06:42:58.502ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30482025-04-30T06:42:58.508ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30492025-04-30T06:42:58.511ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30502025-04-30T06:42:58.517ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30512025-04-30T06:42:58.526ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30522025-04-30T06:42:58.537ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30532025-04-30T06:42:58.545ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30542025-04-30T06:42:58.549ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30552025-04-30T06:42:58.558ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30562025-04-30T06:42:58.562ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30572025-04-30T06:42:58.567ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30582025-04-30T06:42:58.581ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30592025-04-30T06:42:58.588ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30602025-04-30T06:42:58.601ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30612025-04-30T06:42:58.606ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30622025-04-30T06:42:58.611ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30632025-04-30T06:42:58.616ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30642025-04-30T06:42:58.623ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30652025-04-30T06:42:58.628ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30662025-04-30T06:42:58.632ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30672025-04-30T06:42:58.636ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30682025-04-30T06:42:58.640ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30692025-04-30T06:42:58.643ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30702025-04-30T06:42:58.646ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30712025-04-30T06:42:58.648ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30722025-04-30T06:42:58.652ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30732025-04-30T06:42:58.655ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30742025-04-30T06:42:58.659ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30752025-04-30T06:42:58.662ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30762025-04-30T06:42:58.666ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30772025-04-30T06:42:58.671ZINFOtest_snapshot_expunge (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = f2d4d5c2-3547-4831-8529-796764280fb6 version = 14
30782025-04-30T06:42:58.671ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30792025-04-30T06:42:58.677ZDEBGtest_snapshot_expunge (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 07a65907-a22d-473d-98a6-8b061d85aca7
30802025-04-30T06:42:58.678ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:64353 method = GET path = /info versions = all
30812025-04-30T06:42:58.678ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:64353 method = GET path = /producers versions = all
30822025-04-30T06:42:58.679ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:64353 method = DELETE path = /producers/{producer_id} versions = all
30832025-04-30T06:42:58.679ZDEBGtest_snapshot_expunge (dropshot): registered endpoint local_addr = [::1]:64353 method = GET path = /producers/{producer_id} versions = all
30842025-04-30T06:42:58.679ZINFOtest_snapshot_expunge (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:64353
30852025-04-30T06:42:58.679ZDEBGtest_snapshot_expunge (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:64353
30862025-04-30T06:42:58.681ZDEBGtest_snapshot_expunge (oximeter): registered USDT probes
30872025-04-30T06:42:58.681ZDEBGtest_snapshot_expunge (oximeter): contacting nexus
30882025-04-30T06:42:59.387ZDEBGtest_snapshot_expunge (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:41836/metrics/collectors
30892025-04-30T06:42:59.387ZDEBGtest_snapshot_expunge (producer-registration-task): will register with Nexus at 127.0.0.1:41836
30902025-04-30T06:42:59.500ZDEBGtest_snapshot_expunge (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:41836/metrics/producers
30912025-04-30T06:42:59.503ZINFOtest_snapshot_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:41836 remote_addr = 127.0.0.1:57068
30922025-04-30T06:42:59.503ZINFOtest_snapshot_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:41836 remote_addr = 127.0.0.1:35098
30932025-04-30T06:42:59.506ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:35098 req_id = e0a0346a-27e9-4e20-b617-8be6a5b7a041 uri = /metrics/producers
30942025-04-30T06:42:59.507ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:35098 req_id = e0a0346a-27e9-4e20-b617-8be6a5b7a041 resource = Database uri = /metrics/producers
30952025-04-30T06:42:59.507ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:35098 req_id = e0a0346a-27e9-4e20-b617-8be6a5b7a041 roles = RoleSet { roles: {} } uri = /metrics/producers
30962025-04-30T06:42:59.509ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:35098 req_id = e0a0346a-27e9-4e20-b617-8be6a5b7a041 resource = Database result = Ok(()) uri = /metrics/producers
30972025-04-30T06:42:59.529ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:57068 req_id = 1f06c064-ce53-451c-b76a-9537beba20e5 uri = /metrics/collectors
30982025-04-30T06:42:59.529ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:57068 req_id = 1f06c064-ce53-451c-b76a-9537beba20e5 resource = Database uri = /metrics/collectors
30992025-04-30T06:42:59.529ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:57068 req_id = 1f06c064-ce53-451c-b76a-9537beba20e5 roles = RoleSet { roles: {} } uri = /metrics/collectors
31002025-04-30T06:42:59.530ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:57068 req_id = 1f06c064-ce53-451c-b76a-9537beba20e5 resource = Database result = Ok(()) uri = /metrics/collectors
31012025-04-30T06:42:59.541ZINFOtest_snapshot_expunge (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 34734 local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:35098 req_id = e0a0346a-27e9-4e20-b617-8be6a5b7a041 response_code = 503 uri = /metrics/producers
31022025-04-30T06:42:59.541ZINFOtest_snapshot_expunge (nexus): registered new oximeter metric collection server address = [::1]:64353 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31032025-04-30T06:42:59.541ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 12686 local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:57068 req_id = 1f06c064-ce53-451c-b76a-9537beba20e5 response_code = 204 uri = /metrics/collectors
31042025-04-30T06:42:59.542ZDEBGtest_snapshot_expunge (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "e0a0346a-27e9-4e20-b617-8be6a5b7a041", "content-length": "133", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
31052025-04-30T06:42:59.544ZWARNtest_snapshot_expunge (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 1.767386812s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"e0a0346a-27e9-4e20-b617-8be6a5b7a041\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:42:59 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"e0a0346a-27e9-4e20-b617-8be6a5b7a041\\" }" file = oximeter/producer/src/lib.rs:371
31062025-04-30T06:42:59.544ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors", status: 204, headers: {"x-request-id": "1f06c064-ce53-451c-b76a-9537beba20e5", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
31072025-04-30T06:42:59.545ZINFOtest_snapshot_expunge (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31082025-04-30T06:42:59.545ZDEBGtest_snapshot_expunge (init_with_steps): Running step start_producer_server
31092025-04-30T06:42:59.545ZDEBGtest_snapshot_expunge: Starting test metric Producer Server
31102025-04-30T06:42:59.550ZINFOtest_snapshot_expunge: registering Oso class class = Action
31112025-04-30T06:42:59.550ZINFOtest_snapshot_expunge: registering Oso class class = AnyActor
31122025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = AuthenticatedActor
31132025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = BlueprintConfig
31142025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Database
31152025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = DnsConfig
31162025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Fleet
31172025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Inventory
31182025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = IpPoolList
31192025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = ConsoleSessionList
31202025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = DeviceAuthRequestList
31212025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = SiloCertificateList
31222025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = SiloIdentityProviderList
31232025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = SiloUserList
31242025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = TargetReleaseConfig
31252025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = WebhookEventClassList
31262025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Project
31272025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Disk
31282025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Snapshot
31292025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = ProjectImage
31302025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = AffinityGroup
31312025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = AntiAffinityGroup
31322025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Instance
31332025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = IpPool
31342025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = InstanceNetworkInterface
31352025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Vpc
31362025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = VpcRouter
31372025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = InternetGateway
31382025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = InternetGatewayIpPool
31392025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = InternetGatewayIpAddress
31402025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = RouterRoute
31412025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = VpcSubnet
31422025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = FloatingIp
31432025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Image
31442025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = SiloImage
31452025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = AddressLot
31462025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Blueprint
31472025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = LoopbackAddress
31482025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Certificate
31492025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = ConsoleSession
31502025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = DeviceAuthRequest
31512025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = DeviceAccessToken
31522025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = PhysicalDisk
31532025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = Rack
31542025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = RoleBuiltin
31552025-04-30T06:42:59.551ZINFOtest_snapshot_expunge: registering Oso class class = SshKey
31562025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = Silo
31572025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = SiloUser
31582025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = SiloGroup
31592025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = SupportBundle
31602025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = IdentityProvider
31612025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = SamlIdentityProvider
31622025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = Sled
31632025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = TufRepo
31642025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = TufArtifact
31652025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = WebhookEvent
31662025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = WebhookReceiver
31672025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = WebhookSecret
31682025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = Zpool
31692025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = Service
31702025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: registering Oso class class = UserBuiltin
31712025-04-30T06:42:59.552ZINFOtest_snapshot_expunge: full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
31722025-04-30T06:42:59.576ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64060 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31732025-04-30T06:42:59.577ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64060 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31742025-04-30T06:42:59.577ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64060 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31752025-04-30T06:42:59.577ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64060 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31762025-04-30T06:42:59.577ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64060 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31772025-04-30T06:42:59.577ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64060 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31782025-04-30T06:42:59.577ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64060 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31792025-04-30T06:42:59.577ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64060 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31802025-04-30T06:42:59.577ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64060 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31812025-04-30T06:42:59.577ZINFOtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:64060 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31822025-04-30T06:42:59.579ZDEBGtest_snapshot_expunge (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:64060 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31832025-04-30T06:42:59.579ZINFOtest_snapshot_expunge (SledAgent): Created Simulated Crucible Server address = [::1]:64060 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31842025-04-30T06:42:59.579ZINFOtest_snapshot_expunge (nexus): inserting test physical disk model = test-model physical_disk_id = ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8 serial = totally-unique-serial: ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31852025-04-30T06:42:59.579ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31862025-04-30T06:42:59.579ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
31872025-04-30T06:42:59.580ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31882025-04-30T06:42:59.582ZINFOtest_snapshot_expunge (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31892025-04-30T06:42:59.663ZINFOtest_snapshot_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:41836 remote_addr = 127.0.0.1:41716
31902025-04-30T06:42:59.664ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31912025-04-30T06:42:59.665ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31922025-04-30T06:42:59.667ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31932025-04-30T06:42:59.667ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
31942025-04-30T06:42:59.668ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31952025-04-30T06:42:59.669ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:41716 req_id = 93d16702-9355-418d-a859-5ebf99042ec5 uri = /metrics/producers
31962025-04-30T06:42:59.670ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 31793975-499f-4f18-ad9a-1e409a2463d1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31972025-04-30T06:42:59.674ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:41716 req_id = 93d16702-9355-418d-a859-5ebf99042ec5 resource = Database uri = /metrics/producers
31982025-04-30T06:42:59.674ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:41716 req_id = 93d16702-9355-418d-a859-5ebf99042ec5 roles = RoleSet { roles: {} } uri = /metrics/producers
31992025-04-30T06:42:59.674ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:41716 req_id = 93d16702-9355-418d-a859-5ebf99042ec5 resource = Database result = Ok(()) uri = /metrics/producers
32002025-04-30T06:42:59.675ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 31793975-499f-4f18-ad9a-1e409a2463d1 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
32012025-04-30T06:42:59.675ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 31793975-499f-4f18-ad9a-1e409a2463d1 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
32022025-04-30T06:42:59.675ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 31793975-499f-4f18-ad9a-1e409a2463d1 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
32032025-04-30T06:42:59.679ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32042025-04-30T06:42:59.679ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32052025-04-30T06:42:59.679ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32062025-04-30T06:42:59.679ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32072025-04-30T06:42:59.679ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32082025-04-30T06:42:59.683ZTRACtest_snapshot_expunge (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32092025-04-30T06:42:59.683ZTRACtest_snapshot_expunge (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32102025-04-30T06:42:59.688ZINFOtest_snapshot_expunge (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
32112025-04-30T06:42:59.689ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 19411 local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:41716 req_id = 93d16702-9355-418d-a859-5ebf99042ec5 response_code = 201 uri = /metrics/producers
32122025-04-30T06:42:59.689ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 19593 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 31793975-499f-4f18-ad9a-1e409a2463d1 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
32132025-04-30T06:42:59.693ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "31793975-499f-4f18-ad9a-1e409a2463d1", "content-length": "281", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
32142025-04-30T06:42:59.700ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
32152025-04-30T06:42:59.703ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32162025-04-30T06:42:59.707ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32172025-04-30T06:42:59.708ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 26c6b781-42a6-4115-8486-1610936747ab uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
32182025-04-30T06:42:59.712ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 26c6b781-42a6-4115-8486-1610936747ab resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
32192025-04-30T06:42:59.712ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 26c6b781-42a6-4115-8486-1610936747ab roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
32202025-04-30T06:42:59.712ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 26c6b781-42a6-4115-8486-1610936747ab resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
32212025-04-30T06:42:59.721ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 13989 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 26c6b781-42a6-4115-8486-1610936747ab response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
32222025-04-30T06:42:59.722ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "26c6b781-42a6-4115-8486-1610936747ab", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
32232025-04-30T06:42:59.722ZDEBGtest_snapshot_expunge (oximeter-agent): registered new metric producer address = [::1]:47657 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
32242025-04-30T06:42:59.779ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
32252025-04-30T06:42:59.784ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 1 n_pruned_tasks = 0
32262025-04-30T06:42:59.836ZINFOtest_snapshot_expunge (nexus): upserting zpool physical_disk_id = ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 3db19815-011f-4378-bd21-57ee0c4df822
32272025-04-30T06:42:59.836ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32282025-04-30T06:42:59.836ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32292025-04-30T06:42:59.836ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32302025-04-30T06:42:59.840ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32312025-04-30T06:42:59.840ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32322025-04-30T06:42:59.840ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32332025-04-30T06:42:59.840ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32342025-04-30T06:42:59.840ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32352025-04-30T06:42:59.845ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32362025-04-30T06:42:59.849ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32372025-04-30T06:42:59.863ZINFOtest_snapshot_expunge (nexus): upserting Crucible dataset dataset_id = a639acdc-dce2-44d8-a2af-a44d0b009fe4 zpool_id = 3db19815-011f-4378-bd21-57ee0c4df822
32382025-04-30T06:42:59.875ZINFOtest_snapshot_expunge (nexus): inserting test physical disk model = test-model physical_disk_id = ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8 serial = totally-unique-serial: ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
32392025-04-30T06:42:59.882ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32402025-04-30T06:42:59.882ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32412025-04-30T06:42:59.883ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32422025-04-30T06:42:59.887ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8 (physical_disk), lookup_type: ById(ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8) }
32432025-04-30T06:42:59.887ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32442025-04-30T06:42:59.887ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32452025-04-30T06:42:59.887ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32462025-04-30T06:42:59.890ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32472025-04-30T06:42:59.893ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32482025-04-30T06:42:59.906ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8 (physical_disk), lookup_type: ById(ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8) } result = Ok(())
32492025-04-30T06:42:59.906ZINFOtest_snapshot_expunge (nexus): upserting zpool physical_disk_id = ddb76267-0ec7-4e45-ad56-36c1bdfe3ff8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 3db19815-011f-4378-bd21-57ee0c4df822
32502025-04-30T06:42:59.907ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32512025-04-30T06:42:59.907ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32522025-04-30T06:42:59.907ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32532025-04-30T06:42:59.908ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32542025-04-30T06:42:59.908ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32552025-04-30T06:42:59.908ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32562025-04-30T06:42:59.908ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32572025-04-30T06:42:59.911ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32582025-04-30T06:42:59.919ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32592025-04-30T06:42:59.923ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32602025-04-30T06:42:59.937ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32612025-04-30T06:42:59.937ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32622025-04-30T06:42:59.937ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32632025-04-30T06:42:59.937ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32642025-04-30T06:42:59.938ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32652025-04-30T06:42:59.941ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32662025-04-30T06:42:59.945ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32672025-04-30T06:42:59.945ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32682025-04-30T06:42:59.945ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32692025-04-30T06:42:59.945ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32702025-04-30T06:42:59.955ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
32712025-04-30T06:42:59.956ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32722025-04-30T06:42:59.956ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32732025-04-30T06:42:59.956ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32742025-04-30T06:42:59.956ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32752025-04-30T06:42:59.956ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
32762025-04-30T06:42:59.957ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
32772025-04-30T06:42:59.957ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32782025-04-30T06:42:59.957ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32792025-04-30T06:42:59.957ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32802025-04-30T06:42:59.957ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32812025-04-30T06:42:59.960ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32822025-04-30T06:42:59.964ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
32832025-04-30T06:42:59.964ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32842025-04-30T06:42:59.964ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32852025-04-30T06:42:59.965ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32862025-04-30T06:42:59.965ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32872025-04-30T06:42:59.971ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32882025-04-30T06:42:59.971ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32892025-04-30T06:42:59.971ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32902025-04-30T06:42:59.971ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32912025-04-30T06:42:59.976ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32922025-04-30T06:42:59.976ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32932025-04-30T06:42:59.977ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32942025-04-30T06:42:59.977ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
32952025-04-30T06:42:59.977ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
32962025-04-30T06:42:59.980ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f5 00 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50475 req_id = e2f99a81-6408-4808-b866-b94496d4a80a
32972025-04-30T06:42:59.982ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62720,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50475 req_id = e2f99a81-6408-4808-b866-b94496d4a80a
32982025-04-30T06:42:59.983ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
32992025-04-30T06:42:59.983ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
33002025-04-30T06:42:59.983ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50475 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e2f99a81-6408-4808-b866-b94496d4a80a
33012025-04-30T06:42:59.988ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4244, tv_nsec: 787469628 } })
33022025-04-30T06:42:59.990ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
33032025-04-30T06:42:59.991ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e 87 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59935 req_id = 06040aff-f27f-4508-acb5-caca9d863932
33042025-04-30T06:42:59.991ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40583,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59935 req_id = 06040aff-f27f-4508-acb5-caca9d863932
33052025-04-30T06:42:59.991ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
33062025-04-30T06:42:59.991ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59935 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 06040aff-f27f-4508-acb5-caca9d863932
33072025-04-30T06:43:00.054ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
33082025-04-30T06:43:00.066ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: bb cb 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54741 req_id = b2cb2843-a352-41bd-bd59-6c5243942699
33092025-04-30T06:43:00.067ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48075,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54741 req_id = b2cb2843-a352-41bd-bd59-6c5243942699
33102025-04-30T06:43:00.067ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
33112025-04-30T06:43:00.067ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48075, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54741 req_id = b2cb2843-a352-41bd-bd59-6c5243942699
33122025-04-30T06:43:00.068ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
33132025-04-30T06:43:00.068ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
33142025-04-30T06:43:00.068ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
33152025-04-30T06:43:00.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "01f863d5-f3a9-4bbf-b5d3-499aadd664c1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33162025-04-30T06:43:00.087ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
33172025-04-30T06:43:00.100ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d53f122f-36c6-4c5e-be51-92928a286cc1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33182025-04-30T06:43:00.100ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
33192025-04-30T06:43:00.105ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae320082-bfdf-4736-a05e-6b2c26bb92df", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33202025-04-30T06:43:00.105ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
33212025-04-30T06:43:00.109ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af5f8aa2-4b3e-4572-8689-98f63603137d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33222025-04-30T06:43:00.109ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
33232025-04-30T06:43:00.112ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1201ffe1-6730-41c3-b12e-f09de9ea8b7f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33242025-04-30T06:43:00.112ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
33252025-04-30T06:43:00.115ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73a4a33f-cb96-427c-8607-5b5f32ae619a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33262025-04-30T06:43:00.115ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
33272025-04-30T06:43:00.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f033b6da-5437-4283-ab60-1029c3430892", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33282025-04-30T06:43:00.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
33292025-04-30T06:43:00.124ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1962cbda-c509-44fa-a8c5-3d469ccbd56a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33302025-04-30T06:43:00.125ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
33312025-04-30T06:43:00.126ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
33322025-04-30T06:43:00.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "feede402-cd6a-4e89-b6ca-183ec6f8ebcc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33332025-04-30T06:43:00.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
33342025-04-30T06:43:00.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ebfd014-5d6c-4375-9a75-914c5c35dbff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33352025-04-30T06:43:00.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
33362025-04-30T06:43:00.131ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34c64e19-3dd4-4aee-b0ae-ed768470ee38", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33372025-04-30T06:43:00.131ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
33382025-04-30T06:43:00.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3df5d5e1-1d2b-4369-a689-2f4ecdce28db", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33392025-04-30T06:43:00.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
33402025-04-30T06:43:00.134ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3707f92b-2ecc-4952-91c5-9da458c5312f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33412025-04-30T06:43:00.134ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
33422025-04-30T06:43:00.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e3c1757-777c-470c-b059-a1e49db210b4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33432025-04-30T06:43:00.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
33442025-04-30T06:43:00.138ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1169329c-2dc3-43ef-941e-e3721732d8ce", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33452025-04-30T06:43:00.138ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
33462025-04-30T06:43:00.141ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcb4b42a-a1d6-4d80-8842-99b12c1af7a3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33472025-04-30T06:43:00.141ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
33482025-04-30T06:43:00.144ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1303dc3b-d8ec-476d-af43-4192a2374470", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33492025-04-30T06:43:00.144ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
33502025-04-30T06:43:00.146ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84b490bd-338f-4e13-b10c-ff88c921381d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33512025-04-30T06:43:00.146ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
33522025-04-30T06:43:00.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99b6e802-a4f8-4e09-9d62-d0fc36cac97e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33532025-04-30T06:43:00.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
33542025-04-30T06:43:00.150ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53cb2b93-ad1e-4ce4-9237-51b939473ba5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33552025-04-30T06:43:00.150ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
33562025-04-30T06:43:00.151ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2441e959-069d-474e-bce8-c7b5ae8ff0ad", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33572025-04-30T06:43:00.151ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
33582025-04-30T06:43:00.153ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d931ed1-42db-4016-a465-b046b6a28bd7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33592025-04-30T06:43:00.153ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
33602025-04-30T06:43:00.154ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e72ab0f3-b171-4371-8710-008885691309", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33612025-04-30T06:43:00.154ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
33622025-04-30T06:43:00.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95de63b4-97e7-4aa6-9637-212e2399e5c8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33632025-04-30T06:43:00.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
33642025-04-30T06:43:00.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f7f47b8-be3c-429e-965a-8413cdcbb0ea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33652025-04-30T06:43:00.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
33662025-04-30T06:43:00.166ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44067e99-d06a-4b96-ae51-f9e5808cce0b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33672025-04-30T06:43:00.166ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
33682025-04-30T06:43:00.169ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3bfb3a5-c48b-496b-b83a-8434153b16df", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33692025-04-30T06:43:00.169ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
33702025-04-30T06:43:00.173ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c28aae9d-20aa-4fc5-af37-ad746e2f977c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33712025-04-30T06:43:00.173ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
33722025-04-30T06:43:00.176ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cba8c76b-5a88-4070-9476-182126147a66", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33732025-04-30T06:43:00.177ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
33742025-04-30T06:43:00.177ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33752025-04-30T06:43:00.177ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33762025-04-30T06:43:00.177ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33772025-04-30T06:43:00.177ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
33782025-04-30T06:43:00.179ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33792025-04-30T06:43:00.187ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33802025-04-30T06:43:00.194ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33812025-04-30T06:43:00.194ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33822025-04-30T06:43:00.194ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
33832025-04-30T06:43:00.195ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33842025-04-30T06:43:00.197ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2c6d609-3471-40f5-819f-750d29bdb642", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33852025-04-30T06:43:00.198ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
33862025-04-30T06:43:00.200ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33872025-04-30T06:43:00.200ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
33882025-04-30T06:43:00.200ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33892025-04-30T06:43:00.202ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eda7f4cb-142f-4029-a7ab-666afe4879a2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33902025-04-30T06:43:00.202ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
33912025-04-30T06:43:00.209ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a04ddb9e-f6d6-4fbc-88ee-35d42fe5dae8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33922025-04-30T06:43:00.210ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
33932025-04-30T06:43:00.212ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67e974a3-315b-4d21-bf71-38dd378ae664", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33942025-04-30T06:43:00.212ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
33952025-04-30T06:43:00.215ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2909456a-cd99-428a-8776-727f54789983", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33962025-04-30T06:43:00.215ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
33972025-04-30T06:43:00.218ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8849717d-7140-4a69-9097-a56c9243da4f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
33982025-04-30T06:43:00.218ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
33992025-04-30T06:43:00.230ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ca48205-ea95-4df5-b56d-7f4592832ab3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34002025-04-30T06:43:00.231ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
34012025-04-30T06:43:00.236ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "035286dc-bc72-4a47-9d5c-14d3bfe39a00", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34022025-04-30T06:43:00.236ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
34032025-04-30T06:43:00.239ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd34360d-0b6e-4de9-abb6-c93a05f5af8e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34042025-04-30T06:43:00.239ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
34052025-04-30T06:43:00.246ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7582d78-c7b8-43d5-911e-9a5f960dea4e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34062025-04-30T06:43:00.246ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
34072025-04-30T06:43:00.251ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4c6b34d-2896-489f-ae42-77486a786d6c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34082025-04-30T06:43:00.251ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
34092025-04-30T06:43:00.257ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a571b496-cdbb-42d6-909d-fdd1ca92bf67", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34102025-04-30T06:43:00.257ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
34112025-04-30T06:43:00.262ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e31ed1dd-eda3-48f1-a22f-2940ec05a713", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34122025-04-30T06:43:00.262ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
34132025-04-30T06:43:00.264ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
34142025-04-30T06:43:00.266ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ec96bb8-72a6-42f6-a6f6-2665ac393519", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34152025-04-30T06:43:00.266ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
34162025-04-30T06:43:00.267ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d94af38-8b43-48fc-9734-beacaffdcd17", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34172025-04-30T06:43:00.267ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
34182025-04-30T06:43:00.269ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fe11fc1-e026-4a38-937a-e130e5465fa6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34192025-04-30T06:43:00.270ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
34202025-04-30T06:43:00.270ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34212025-04-30T06:43:00.270ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34222025-04-30T06:43:00.270ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34232025-04-30T06:43:00.270ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34242025-04-30T06:43:00.283ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34252025-04-30T06:43:00.286ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
34262025-04-30T06:43:00.286ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
34272025-04-30T06:43:00.286ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34282025-04-30T06:43:00.286ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34292025-04-30T06:43:00.286ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34302025-04-30T06:43:00.287ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34312025-04-30T06:43:00.291ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34322025-04-30T06:43:00.295ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
34332025-04-30T06:43:00.295ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34342025-04-30T06:43:00.295ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34352025-04-30T06:43:00.295ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34362025-04-30T06:43:00.357ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
34372025-04-30T06:43:00.357ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
34382025-04-30T06:43:00.358ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34392025-04-30T06:43:00.358ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34402025-04-30T06:43:00.358ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34412025-04-30T06:43:00.358ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34422025-04-30T06:43:00.359ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34432025-04-30T06:43:00.359ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:40189 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
34442025-04-30T06:43:00.360ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:40189 req_id = 9b671e8f-0364-4e98-af0e-23e0f28b25c1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
34452025-04-30T06:43:00.362ZDEBGtest_snapshot_expunge (producer-registration-task): will register with Nexus at 127.0.0.1:41836
34462025-04-30T06:43:00.411ZDEBGtest_snapshot_expunge (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:41836/metrics/producers
34472025-04-30T06:43:00.411ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 51149 local_addr = [::1]:51119 method = GET remote_addr = [::1]:40189 req_id = 9b671e8f-0364-4e98-af0e-23e0f28b25c1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
34482025-04-30T06:43:00.412ZINFOtest_snapshot_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:41836 remote_addr = 127.0.0.1:52934
34492025-04-30T06:43:00.412ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b671e8f-0364-4e98-af0e-23e0f28b25c1", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
34502025-04-30T06:43:00.414ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
34512025-04-30T06:43:00.414ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
34522025-04-30T06:43:00.414ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
34532025-04-30T06:43:00.415ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34542025-04-30T06:43:00.415ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34552025-04-30T06:43:00.416ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34562025-04-30T06:43:00.416ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34572025-04-30T06:43:00.416ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34582025-04-30T06:43:00.418ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34592025-04-30T06:43:00.428ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34602025-04-30T06:43:00.428ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34612025-04-30T06:43:00.428ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34622025-04-30T06:43:00.429ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34632025-04-30T06:43:00.432ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:52934 req_id = 08ffdfa3-1d6a-43c4-8255-9da12fc4f7b0 uri = /metrics/producers
34642025-04-30T06:43:00.433ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:52934 req_id = 08ffdfa3-1d6a-43c4-8255-9da12fc4f7b0 resource = Database uri = /metrics/producers
34652025-04-30T06:43:00.433ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:52934 req_id = 08ffdfa3-1d6a-43c4-8255-9da12fc4f7b0 roles = RoleSet { roles: {} } uri = /metrics/producers
34662025-04-30T06:43:00.452ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:52934 req_id = 08ffdfa3-1d6a-43c4-8255-9da12fc4f7b0 resource = Database result = Ok(()) uri = /metrics/producers
34672025-04-30T06:43:00.456ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34682025-04-30T06:43:00.456ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34692025-04-30T06:43:00.459ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34702025-04-30T06:43:00.459ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34712025-04-30T06:43:00.466ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34722025-04-30T06:43:00.467ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34732025-04-30T06:43:00.467ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34742025-04-30T06:43:00.467ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34752025-04-30T06:43:00.525ZINFOtest_snapshot_expunge (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
34762025-04-30T06:43:00.525ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 93330 local_addr = 127.0.0.1:41836 method = POST remote_addr = 127.0.0.1:52934 req_id = 08ffdfa3-1d6a-43c4-8255-9da12fc4f7b0 response_code = 201 uri = /metrics/producers
34772025-04-30T06:43:00.527ZDEBGtest_snapshot_expunge (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "08ffdfa3-1d6a-43c4-8255-9da12fc4f7b0", "content-length": "41", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
34782025-04-30T06:43:00.530ZDEBGtest_snapshot_expunge (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
34792025-04-30T06:43:00.596ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
34802025-04-30T06:43:00.626ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
34812025-04-30T06:43:00.627ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
34822025-04-30T06:43:00.627ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 670.665498ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
34832025-04-30T06:43:00.627ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
34842025-04-30T06:43:00.628ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34852025-04-30T06:43:00.628ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34862025-04-30T06:43:00.628ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34872025-04-30T06:43:00.628ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34882025-04-30T06:43:00.629ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34892025-04-30T06:43:00.634ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34902025-04-30T06:43:00.639ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34912025-04-30T06:43:00.639ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34922025-04-30T06:43:00.640ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34932025-04-30T06:43:00.640ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34942025-04-30T06:43:00.652ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34952025-04-30T06:43:00.652ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34962025-04-30T06:43:00.652ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34972025-04-30T06:43:00.652ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34982025-04-30T06:43:00.653ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34992025-04-30T06:43:00.660ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
35002025-04-30T06:43:00.660ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
35012025-04-30T06:43:00.662ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: da 82 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51244 req_id = e52439ee-d71c-471e-8bdd-73d8f7bf85a3
35022025-04-30T06:43:00.662ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55938,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51244 req_id = e52439ee-d71c-471e-8bdd-73d8f7bf85a3
35032025-04-30T06:43:00.662ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
35042025-04-30T06:43:00.663ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
35052025-04-30T06:43:00.663ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51244 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e52439ee-d71c-471e-8bdd-73d8f7bf85a3
35062025-04-30T06:43:00.663ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35072025-04-30T06:43:00.669ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35082025-04-30T06:43:00.669ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35092025-04-30T06:43:00.669ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
35102025-04-30T06:43:00.669ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35112025-04-30T06:43:00.670ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4245, tv_nsec: 469238037 } })
35122025-04-30T06:43:00.670ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
35132025-04-30T06:43:00.676ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 09 f8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59118 req_id = 5ce12125-7317-4854-877b-038a02aa4df9
35142025-04-30T06:43:00.677ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2552,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59118 req_id = 5ce12125-7317-4854-877b-038a02aa4df9
35152025-04-30T06:43:00.677ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
35162025-04-30T06:43:00.677ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59118 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5ce12125-7317-4854-877b-038a02aa4df9
35172025-04-30T06:43:00.736ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
35182025-04-30T06:43:00.737ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35192025-04-30T06:43:00.737ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
35202025-04-30T06:43:00.737ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35212025-04-30T06:43:00.738ZTRACtest_snapshot_expunge (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35222025-04-30T06:43:00.738ZTRACtest_snapshot_expunge (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35232025-04-30T06:43:00.738ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
35242025-04-30T06:43:00.738ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
35252025-04-30T06:43:00.739ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: fe be 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50428 req_id = c2e670ad-f591-4c1a-a588-d5956da8035d
35262025-04-30T06:43:00.739ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65214,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50428 req_id = c2e670ad-f591-4c1a-a588-d5956da8035d
35272025-04-30T06:43:00.740ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
35282025-04-30T06:43:00.740ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65214, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50428 req_id = c2e670ad-f591-4c1a-a588-d5956da8035d
35292025-04-30T06:43:00.741ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
35302025-04-30T06:43:00.741ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
35312025-04-30T06:43:00.741ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
35322025-04-30T06:43:00.744ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
35332025-04-30T06:43:00.747ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
35342025-04-30T06:43:00.747ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
35352025-04-30T06:43:00.747ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
35362025-04-30T06:43:00.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c611e512-cf8d-4118-8de8-3aa5827c533f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35372025-04-30T06:43:00.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
35382025-04-30T06:43:00.752ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9474b2f7-60fd-4fb6-8b57-2e33fe77ecbc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35392025-04-30T06:43:00.752ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
35402025-04-30T06:43:00.754ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a1eb996-fba5-400f-9246-76f539ec4289", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35412025-04-30T06:43:00.754ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
35422025-04-30T06:43:00.758ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d165de32-ea16-479e-9533-903c5aafc862", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35432025-04-30T06:43:00.758ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
35442025-04-30T06:43:00.760ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02d4a591-d1b3-4acb-a0c5-5ebc491a77a2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35452025-04-30T06:43:00.761ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
35462025-04-30T06:43:00.765ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d53cea0c-a03f-4c36-b38a-5b9e02e125d0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35472025-04-30T06:43:00.765ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
35482025-04-30T06:43:00.799ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
35492025-04-30T06:43:00.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4df3021-5f8d-41f2-a813-589aa4a0ace0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35502025-04-30T06:43:00.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
35512025-04-30T06:43:00.807ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ce5ffc7-f6c4-413d-95ef-5769a1763978", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35522025-04-30T06:43:00.807ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
35532025-04-30T06:43:00.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d23cc66-bfe9-4b47-a16b-9956e4dc6586", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35542025-04-30T06:43:00.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
35552025-04-30T06:43:00.811ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b23ca24-ba37-4ac1-8d9a-0b4210b4d6d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35562025-04-30T06:43:00.812ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
35572025-04-30T06:43:00.813ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4498097-7e5e-4008-b0dc-4e163a022d8f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35582025-04-30T06:43:00.813ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
35592025-04-30T06:43:00.815ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13f44775-811b-4d43-aa16-8a041c64e811", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35602025-04-30T06:43:00.815ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
35612025-04-30T06:43:00.818ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0610722c-7166-45a5-bb12-483632840b22", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35622025-04-30T06:43:00.818ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
35632025-04-30T06:43:00.821ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d51c373d-1121-468d-9e8d-01f144d9d523", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35642025-04-30T06:43:00.822ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
35652025-04-30T06:43:00.825ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6662c8d5-79bc-4710-9c1a-97fbc0ea8284", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35662025-04-30T06:43:00.825ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
35672025-04-30T06:43:00.827ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6888799-b3e0-4ace-85af-bd772cc9deb8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35682025-04-30T06:43:00.828ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
35692025-04-30T06:43:00.836ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f0c1932-1bc1-4cd8-9c1e-f726072d37dd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35702025-04-30T06:43:00.836ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
35712025-04-30T06:43:00.844ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fab1b12-7f74-417e-b682-50a31088f187", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35722025-04-30T06:43:00.844ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
35732025-04-30T06:43:00.853ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35742025-04-30T06:43:00.854ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35752025-04-30T06:43:00.854ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35762025-04-30T06:43:00.854ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
35772025-04-30T06:43:00.855ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35782025-04-30T06:43:00.856ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "683aa7ca-5217-494f-a021-a95b9977f05a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35792025-04-30T06:43:00.856ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
35802025-04-30T06:43:00.858ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd96fe6a-14db-4115-845a-cffd60d8cb31", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35812025-04-30T06:43:00.859ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
35822025-04-30T06:43:00.860ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35832025-04-30T06:43:00.867ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35842025-04-30T06:43:00.867ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35852025-04-30T06:43:00.867ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
35862025-04-30T06:43:00.867ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35872025-04-30T06:43:00.871ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35882025-04-30T06:43:00.871ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
35892025-04-30T06:43:00.872ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35902025-04-30T06:43:00.872ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cbd465c-dc69-423a-8e29-e1ef9910977e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35912025-04-30T06:43:00.872ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
35922025-04-30T06:43:00.874ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d914ab2f-9b4d-4377-8f58-f7dd8631afdb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35932025-04-30T06:43:00.875ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
35942025-04-30T06:43:00.877ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50325605-c1db-4381-888c-5d82d2ce24fc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35952025-04-30T06:43:00.878ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
35962025-04-30T06:43:00.880ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c32c18e-a4d4-4c5e-a02d-1083491c248d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35972025-04-30T06:43:00.880ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
35982025-04-30T06:43:00.883ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4902322-24fc-4876-8678-5f259f52057e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
35992025-04-30T06:43:00.883ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
36002025-04-30T06:43:00.886ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7699c589-a201-454a-b3ee-7d6a2cd6ad08", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36012025-04-30T06:43:00.886ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
36022025-04-30T06:43:00.891ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ae964e9-33a9-4a36-9e29-937d69abc642", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36032025-04-30T06:43:00.891ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
36042025-04-30T06:43:00.894ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "042b9947-b792-4d67-b57c-6965a13e78dc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36052025-04-30T06:43:00.894ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
36062025-04-30T06:43:00.896ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cbb775f-9447-4b15-b254-c8ddf3f769f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36072025-04-30T06:43:00.896ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
36082025-04-30T06:43:00.902ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8c0904a-b414-4e3d-8a9a-14859dc91c8a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36092025-04-30T06:43:00.902ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
36102025-04-30T06:43:00.902ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
36112025-04-30T06:43:00.904ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88433c3d-7707-4b0f-b9f7-b5bc8bd8dbdb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36122025-04-30T06:43:00.904ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
36132025-04-30T06:43:00.906ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e649d511-d84d-4390-b7d9-4c4d26d8ebf5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36142025-04-30T06:43:00.906ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
36152025-04-30T06:43:00.907ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b467709-e0c8-4c1e-b4a8-fb77ee63299b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36162025-04-30T06:43:00.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
36172025-04-30T06:43:00.909ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "780a5eeb-a1c4-421f-b0aa-7fdcfc3bb77c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36182025-04-30T06:43:00.910ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
36192025-04-30T06:43:00.911ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f646faee-deac-4ee9-b7d4-801424dd9c39", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36202025-04-30T06:43:00.912ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
36212025-04-30T06:43:00.915ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cf1f51a-f56e-44ac-87f6-9ff7a355bc7c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36222025-04-30T06:43:00.915ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
36232025-04-30T06:43:00.918ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03741cc6-cea6-4e77-acf7-fef97f808519", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36242025-04-30T06:43:00.919ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
36252025-04-30T06:43:00.921ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38b51102-78b6-4f57-9347-6d363c76b4d1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36262025-04-30T06:43:00.921ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
36272025-04-30T06:43:00.925ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6defb967-a5fa-4bc4-b2a2-2674ed75432c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36282025-04-30T06:43:00.925ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
36292025-04-30T06:43:00.929ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5afa240-52c0-4f2a-a47c-c9a9abcf8820", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36302025-04-30T06:43:00.929ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
36312025-04-30T06:43:00.931ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab7eca06-bd85-467e-bee9-4624fb91544a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36322025-04-30T06:43:00.931ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
36332025-04-30T06:43:00.933ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6c8e4c5-f649-4194-a20f-05a145dd34b1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36342025-04-30T06:43:00.934ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
36352025-04-30T06:43:00.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35b9a3d7-abbb-4724-b4ec-fa3a3f04c242", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36362025-04-30T06:43:00.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
36372025-04-30T06:43:00.938ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "008e2d88-5024-4ebe-9a14-1b0efc9c4e59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36382025-04-30T06:43:00.938ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
36392025-04-30T06:43:00.942ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96a54ad3-9d5d-4033-a8f8-91e1ed418f19", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36402025-04-30T06:43:00.942ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
36412025-04-30T06:43:00.942ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36422025-04-30T06:43:00.942ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36432025-04-30T06:43:00.942ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36442025-04-30T06:43:00.945ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36452025-04-30T06:43:00.948ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36462025-04-30T06:43:00.954ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
36472025-04-30T06:43:00.954ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
36482025-04-30T06:43:00.954ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36492025-04-30T06:43:00.954ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36502025-04-30T06:43:00.954ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36512025-04-30T06:43:00.955ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36522025-04-30T06:43:00.956ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36532025-04-30T06:43:00.956ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36542025-04-30T06:43:00.956ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36552025-04-30T06:43:00.956ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
36562025-04-30T06:43:00.957ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36572025-04-30T06:43:00.958ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36582025-04-30T06:43:00.962ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
36592025-04-30T06:43:00.962ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36602025-04-30T06:43:00.962ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36612025-04-30T06:43:00.963ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36622025-04-30T06:43:00.965ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36632025-04-30T06:43:00.973ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36642025-04-30T06:43:00.973ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36652025-04-30T06:43:00.973ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
36662025-04-30T06:43:00.974ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36672025-04-30T06:43:01.038ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
36682025-04-30T06:43:01.038ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
36692025-04-30T06:43:01.039ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:51159 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
36702025-04-30T06:43:01.041ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36712025-04-30T06:43:01.041ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
36722025-04-30T06:43:01.043ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36732025-04-30T06:43:01.044ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:51159 req_id = f5e06c23-b8de-42cf-a488-bf5f59697369 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
36742025-04-30T06:43:01.046ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 1287 local_addr = [::1]:51119 method = GET remote_addr = [::1]:51159 req_id = f5e06c23-b8de-42cf-a488-bf5f59697369 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
36752025-04-30T06:43:01.047ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5e06c23-b8de-42cf-a488-bf5f59697369", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
36762025-04-30T06:43:01.048ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36772025-04-30T06:43:01.048ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36782025-04-30T06:43:01.049ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
36792025-04-30T06:43:01.050ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36802025-04-30T06:43:01.050ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36812025-04-30T06:43:01.050ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36822025-04-30T06:43:01.050ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36832025-04-30T06:43:01.051ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36842025-04-30T06:43:01.059ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36852025-04-30T06:43:01.063ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36862025-04-30T06:43:01.064ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36872025-04-30T06:43:01.064ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36882025-04-30T06:43:01.065ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36892025-04-30T06:43:01.122ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
36902025-04-30T06:43:01.176ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36912025-04-30T06:43:01.176ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36922025-04-30T06:43:01.176ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36932025-04-30T06:43:01.176ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
36942025-04-30T06:43:01.177ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36952025-04-30T06:43:01.197ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36962025-04-30T06:43:01.210ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36972025-04-30T06:43:01.210ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36982025-04-30T06:43:01.210ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
36992025-04-30T06:43:01.212ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37002025-04-30T06:43:01.216ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
37012025-04-30T06:43:01.216ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
37022025-04-30T06:43:01.216ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 588.687531ms iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
37032025-04-30T06:43:01.216ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
37042025-04-30T06:43:01.216ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
37052025-04-30T06:43:01.216ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37062025-04-30T06:43:01.216ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37072025-04-30T06:43:01.216ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37082025-04-30T06:43:01.217ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37092025-04-30T06:43:01.219ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37102025-04-30T06:43:01.219ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
37112025-04-30T06:43:01.220ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37122025-04-30T06:43:01.224ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37132025-04-30T06:43:01.229ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
37142025-04-30T06:43:01.229ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37152025-04-30T06:43:01.229ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37162025-04-30T06:43:01.230ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37172025-04-30T06:43:01.237ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
37182025-04-30T06:43:01.237ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
37192025-04-30T06:43:01.238ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e1 ad 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63976 req_id = 5dd681c0-3f05-449c-8bff-a88ddf18a8a8
37202025-04-30T06:43:01.238ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57773,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63976 req_id = 5dd681c0-3f05-449c-8bff-a88ddf18a8a8
37212025-04-30T06:43:01.238ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
37222025-04-30T06:43:01.241ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
37232025-04-30T06:43:01.241ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63976 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5dd681c0-3f05-449c-8bff-a88ddf18a8a8
37242025-04-30T06:43:01.241ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4246, tv_nsec: 40747542 } })
37252025-04-30T06:43:01.243ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
37262025-04-30T06:43:01.243ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 42 d3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e B...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59921 req_id = e0e8803b-f25c-45d4-be35-837e3634aac8
37272025-04-30T06:43:01.243ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17107,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59921 req_id = e0e8803b-f25c-45d4-be35-837e3634aac8
37282025-04-30T06:43:01.243ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
37292025-04-30T06:43:01.243ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59921 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e0e8803b-f25c-45d4-be35-837e3634aac8
37302025-04-30T06:43:01.312ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
37312025-04-30T06:43:01.314ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 59 bf 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Y............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63888 req_id = fd0d1a2f-f42b-4ae4-891e-21ba7a1b7915
37322025-04-30T06:43:01.314ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22975,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63888 req_id = fd0d1a2f-f42b-4ae4-891e-21ba7a1b7915
37332025-04-30T06:43:01.314ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
37342025-04-30T06:43:01.314ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22975, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63888 req_id = fd0d1a2f-f42b-4ae4-891e-21ba7a1b7915
37352025-04-30T06:43:01.315ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
37362025-04-30T06:43:01.315ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
37372025-04-30T06:43:01.315ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
37382025-04-30T06:43:01.317ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "134f1a65-1fe5-48fa-a064-22194e585b1a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37392025-04-30T06:43:01.317ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
37402025-04-30T06:43:01.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ea0e782-49b1-422c-a348-b88a586b30bc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37412025-04-30T06:43:01.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
37422025-04-30T06:43:01.322ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c72d0f2c-ae43-4fcc-a635-90cb3f3890c7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37432025-04-30T06:43:01.322ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
37442025-04-30T06:43:01.324ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b04688e-2e44-41e2-aae3-c6f05f344626", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37452025-04-30T06:43:01.324ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
37462025-04-30T06:43:01.326ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82a4d32c-2a45-4785-a391-1b55cc9233a0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37472025-04-30T06:43:01.326ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
37482025-04-30T06:43:01.329ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d65dd03d-2352-4614-85cf-09780fac262e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37492025-04-30T06:43:01.329ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
37502025-04-30T06:43:01.331ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b888936-94bd-4629-8bf2-801403fc603f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37512025-04-30T06:43:01.331ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
37522025-04-30T06:43:01.333ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b399cc15-d6c7-49dc-bc9e-4e80473f0ecd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37532025-04-30T06:43:01.333ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
37542025-04-30T06:43:01.334ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87998cd6-61bc-4227-b83b-5c7dd55887f7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37552025-04-30T06:43:01.334ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
37562025-04-30T06:43:01.335ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72902b2e-8284-4dc5-9162-e94aeb952792", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37572025-04-30T06:43:01.335ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
37582025-04-30T06:43:01.336ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45cb4ac9-0563-45c4-aecf-ebdce07086df", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37592025-04-30T06:43:01.336ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
37602025-04-30T06:43:01.337ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09562169-4643-44b1-9b35-64e2d02712d9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37612025-04-30T06:43:01.337ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
37622025-04-30T06:43:01.338ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65b5f16e-fc46-40e8-b77b-bd6e97b741e0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37632025-04-30T06:43:01.338ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
37642025-04-30T06:43:01.340ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80847caf-feca-4c1d-a39d-840e034554c3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37652025-04-30T06:43:01.340ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
37662025-04-30T06:43:01.341ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5828126-cbfe-4740-890f-6cdacedf79de", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37672025-04-30T06:43:01.342ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
37682025-04-30T06:43:01.343ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a02db43f-3c9f-4066-9d3b-b33225a328ad", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37692025-04-30T06:43:01.343ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
37702025-04-30T06:43:01.345ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1653ad83-fd0a-4fff-bd86-8d3810c672c0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37712025-04-30T06:43:01.345ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
37722025-04-30T06:43:01.347ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d19aa230-e032-4a5d-9dcb-6a8ded24ba44", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37732025-04-30T06:43:01.347ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
37742025-04-30T06:43:01.349ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5f2d56a-a4e1-4c20-9aee-59ec7c620069", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37752025-04-30T06:43:01.349ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
37762025-04-30T06:43:01.350ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b49bc7d9-8418-405c-a669-63d5359d414e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37772025-04-30T06:43:01.350ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
37782025-04-30T06:43:01.351ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94f85bbb-1964-40bc-b84e-e5362c64e7e1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37792025-04-30T06:43:01.351ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
37802025-04-30T06:43:01.354ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2812e5cd-3ca3-4d7a-822a-42febf5a53de", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37812025-04-30T06:43:01.354ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
37822025-04-30T06:43:01.355ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
37832025-04-30T06:43:01.356ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14d83204-0e4b-40d0-a095-8e1e41805929", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37842025-04-30T06:43:01.356ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
37852025-04-30T06:43:01.358ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d51ddd39-7005-41d7-a218-509e1799dd07", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37862025-04-30T06:43:01.358ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
37872025-04-30T06:43:01.370ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a7f8a50-29ec-4780-97a6-f06c66cdcd68", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37882025-04-30T06:43:01.371ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
37892025-04-30T06:43:01.374ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d1b7b9f-84fe-401f-b2df-b1bb2e1170ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37902025-04-30T06:43:01.374ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
37912025-04-30T06:43:01.376ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d1c0ff5-6425-4d15-afce-4a676f01f3ff", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37922025-04-30T06:43:01.376ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
37932025-04-30T06:43:01.378ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e45bcf50-f84f-47b4-a419-2f7e90640c09", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37942025-04-30T06:43:01.378ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
37952025-04-30T06:43:01.380ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eabab6e-6bf0-4ae0-89a7-8f1efc74b006", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37962025-04-30T06:43:01.380ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
37972025-04-30T06:43:01.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5430c096-1536-426a-93be-392dfb34f206", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
37982025-04-30T06:43:01.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
37992025-04-30T06:43:01.383ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a53188bf-4214-4421-b840-9edc7b3f1be2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38002025-04-30T06:43:01.383ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
38012025-04-30T06:43:01.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "459eb80f-b2a5-4ebd-ab75-959cbfb240ee", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38022025-04-30T06:43:01.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
38032025-04-30T06:43:01.385ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec63b4c2-0abf-454e-a198-cb98f68e5efc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38042025-04-30T06:43:01.385ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
38052025-04-30T06:43:01.387ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e5a8466-5aff-429a-8f3e-1c48b45487b4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38062025-04-30T06:43:01.387ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
38072025-04-30T06:43:01.388ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "863adad4-ee79-46d7-b0f0-11147a10b006", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38082025-04-30T06:43:01.388ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
38092025-04-30T06:43:01.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b612e96-1ab4-48d9-a47a-911d7b5e70f9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38102025-04-30T06:43:01.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
38112025-04-30T06:43:01.391ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "959e5f20-f146-487d-9d40-c45c314dc38e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38122025-04-30T06:43:01.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
38132025-04-30T06:43:01.393ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b360b236-bf34-4cda-88b3-7a19a4f3eaaf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38142025-04-30T06:43:01.393ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
38152025-04-30T06:43:01.395ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2074714a-3375-4574-ab85-4dd377ccc5a5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38162025-04-30T06:43:01.395ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
38172025-04-30T06:43:01.397ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a317fa31-95f0-41cb-9448-8aa5d3bf929a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38182025-04-30T06:43:01.397ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
38192025-04-30T06:43:01.399ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68f4b29e-c6cb-4e48-906f-a71ada4f3e4a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38202025-04-30T06:43:01.399ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
38212025-04-30T06:43:01.400ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0bb1982-69ba-4fa7-b2e5-0ef5aa7e4b22", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38222025-04-30T06:43:01.400ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
38232025-04-30T06:43:01.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23528a0b-bfb9-47e4-a52f-76147279e4fe", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38242025-04-30T06:43:01.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
38252025-04-30T06:43:01.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccca3acc-8e6c-4c90-a68a-af093dad1688", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38262025-04-30T06:43:01.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
38272025-04-30T06:43:01.406ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38282025-04-30T06:43:01.406ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38292025-04-30T06:43:01.406ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38302025-04-30T06:43:01.406ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
38312025-04-30T06:43:01.411ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38322025-04-30T06:43:01.411ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e077d83-a96f-4ccd-a252-0bdd83c31d7f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
38332025-04-30T06:43:01.411ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
38342025-04-30T06:43:01.412ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38352025-04-30T06:43:01.412ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38362025-04-30T06:43:01.412ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38372025-04-30T06:43:01.412ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38382025-04-30T06:43:01.416ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38392025-04-30T06:43:01.422ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38402025-04-30T06:43:01.422ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38412025-04-30T06:43:01.422ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
38422025-04-30T06:43:01.422ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38432025-04-30T06:43:01.424ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38442025-04-30T06:43:01.430ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38452025-04-30T06:43:01.430ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
38462025-04-30T06:43:01.431ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38472025-04-30T06:43:01.431ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38482025-04-30T06:43:01.431ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38492025-04-30T06:43:01.432ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38502025-04-30T06:43:01.436ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38512025-04-30T06:43:01.436ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
38522025-04-30T06:43:01.437ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38532025-04-30T06:43:01.438ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38542025-04-30T06:43:01.442ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38552025-04-30T06:43:01.442ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38562025-04-30T06:43:01.443ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38572025-04-30T06:43:01.444ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38582025-04-30T06:43:01.473ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
38592025-04-30T06:43:01.524ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38602025-04-30T06:43:01.524ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38612025-04-30T06:43:01.524ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38622025-04-30T06:43:01.525ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
38632025-04-30T06:43:01.525ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38642025-04-30T06:43:01.528ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38652025-04-30T06:43:01.534ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38662025-04-30T06:43:01.534ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38672025-04-30T06:43:01.534ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
38682025-04-30T06:43:01.534ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38692025-04-30T06:43:01.599ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
38702025-04-30T06:43:01.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
38712025-04-30T06:43:01.604ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:64721 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
38722025-04-30T06:43:01.604ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
38732025-04-30T06:43:01.606ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38742025-04-30T06:43:01.607ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38752025-04-30T06:43:01.607ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
38762025-04-30T06:43:01.609ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38772025-04-30T06:43:01.609ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 1ae1a1de-e634-45f8-9842-ece35ce78d23 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38782025-04-30T06:43:01.609ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:64721 req_id = 5b553e9a-e88a-4127-8135-436cb9f190b8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38792025-04-30T06:43:01.610ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 1ae1a1de-e634-45f8-9842-ece35ce78d23 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38802025-04-30T06:43:01.610ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 1ae1a1de-e634-45f8-9842-ece35ce78d23 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38812025-04-30T06:43:01.612ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 1ae1a1de-e634-45f8-9842-ece35ce78d23 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38822025-04-30T06:43:01.613ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 2843 local_addr = [::1]:51119 method = GET remote_addr = [::1]:64721 req_id = 5b553e9a-e88a-4127-8135-436cb9f190b8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38832025-04-30T06:43:01.613ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b553e9a-e88a-4127-8135-436cb9f190b8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
38842025-04-30T06:43:01.616ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38852025-04-30T06:43:01.616ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38862025-04-30T06:43:01.616ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
38872025-04-30T06:43:01.620ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38882025-04-30T06:43:01.620ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38892025-04-30T06:43:01.620ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38902025-04-30T06:43:01.620ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38912025-04-30T06:43:01.621ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38922025-04-30T06:43:01.628ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 19107 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 1ae1a1de-e634-45f8-9842-ece35ce78d23 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38932025-04-30T06:43:01.629ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ae1a1de-e634-45f8-9842-ece35ce78d23", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
38942025-04-30T06:43:01.629ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38952025-04-30T06:43:01.630ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38962025-04-30T06:43:01.637ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38972025-04-30T06:43:01.637ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38982025-04-30T06:43:01.637ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38992025-04-30T06:43:01.641ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39002025-04-30T06:43:01.642ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 74693312-3039-4cbf-a202-28b1492c706a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
39012025-04-30T06:43:01.643ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 74693312-3039-4cbf-a202-28b1492c706a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
39022025-04-30T06:43:01.643ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 74693312-3039-4cbf-a202-28b1492c706a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
39032025-04-30T06:43:01.648ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 74693312-3039-4cbf-a202-28b1492c706a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
39042025-04-30T06:43:01.657ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 14674 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 74693312-3039-4cbf-a202-28b1492c706a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
39052025-04-30T06:43:01.658ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "74693312-3039-4cbf-a202-28b1492c706a", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39062025-04-30T06:43:01.658ZDEBGtest_snapshot_expunge (oximeter-agent): registered new metric producer address = 127.0.0.1:47175 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
39072025-04-30T06:43:01.730ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
39082025-04-30T06:43:01.731ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
39092025-04-30T06:43:01.731ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
39102025-04-30T06:43:01.731ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39112025-04-30T06:43:01.731ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39122025-04-30T06:43:01.733ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
39132025-04-30T06:43:01.733ZDEBGtest_snapshot_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
39142025-04-30T06:43:01.733ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
39152025-04-30T06:43:01.736ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39162025-04-30T06:43:01.736ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
39172025-04-30T06:43:01.738ZTRACtest_snapshot_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
39182025-04-30T06:43:01.741ZDEBGtest_snapshot_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:01.733688526Z } timeseries_name = integration_target:integration_metric
39192025-04-30T06:43:01.741ZDEBGtest_snapshot_expunge (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
39202025-04-30T06:43:01.742ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
39212025-04-30T06:43:01.742ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39222025-04-30T06:43:01.744ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39232025-04-30T06:43:01.744ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39242025-04-30T06:43:01.744ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
39252025-04-30T06:43:01.749ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
39262025-04-30T06:43:01.756ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
39272025-04-30T06:43:01.756ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
39282025-04-30T06:43:01.761ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
39292025-04-30T06:43:01.762ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
39302025-04-30T06:43:01.762ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
39312025-04-30T06:43:01.812ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39322025-04-30T06:43:01.812ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39332025-04-30T06:43:01.812ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39342025-04-30T06:43:01.813ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
39352025-04-30T06:43:01.813ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39362025-04-30T06:43:01.815ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39372025-04-30T06:43:01.824ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39382025-04-30T06:43:01.825ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39392025-04-30T06:43:01.825ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
39402025-04-30T06:43:01.826ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39412025-04-30T06:43:01.832ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
39422025-04-30T06:43:01.848ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
39432025-04-30T06:43:01.849ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
39442025-04-30T06:43:01.849ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 632.418843ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
39452025-04-30T06:43:01.849ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
39462025-04-30T06:43:01.849ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
39472025-04-30T06:43:01.849ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39482025-04-30T06:43:01.849ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39492025-04-30T06:43:01.849ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39502025-04-30T06:43:01.850ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39512025-04-30T06:43:01.850ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39522025-04-30T06:43:01.850ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
39532025-04-30T06:43:01.851ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39542025-04-30T06:43:01.853ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39552025-04-30T06:43:01.873ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
39562025-04-30T06:43:01.873ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39572025-04-30T06:43:01.873ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39582025-04-30T06:43:01.874ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39592025-04-30T06:43:01.908ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
39602025-04-30T06:43:01.908ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
39612025-04-30T06:43:01.915ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 62 b4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 b............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54638 req_id = 3bbef045-b3e2-4ec7-8f44-25062afbd862
39622025-04-30T06:43:01.915ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25268,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54638 req_id = 3bbef045-b3e2-4ec7-8f44-25062afbd862
39632025-04-30T06:43:01.915ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
39642025-04-30T06:43:01.915ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
39652025-04-30T06:43:01.916ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54638 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3bbef045-b3e2-4ec7-8f44-25062afbd862
39662025-04-30T06:43:01.916ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4246, tv_nsec: 715630291 } })
39672025-04-30T06:43:01.916ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
39682025-04-30T06:43:01.917ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 56 fa 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e V...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57991 req_id = c99ccd63-a820-4b19-ad69-b2bb56b88af3
39692025-04-30T06:43:01.917ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22266,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57991 req_id = c99ccd63-a820-4b19-ad69-b2bb56b88af3
39702025-04-30T06:43:01.917ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
39712025-04-30T06:43:01.917ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57991 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c99ccd63-a820-4b19-ad69-b2bb56b88af3
39722025-04-30T06:43:01.973ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
39732025-04-30T06:43:01.975ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e2 e5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59611 req_id = 69c248ea-192e-4003-b05a-4a8cdf65fc6e
39742025-04-30T06:43:01.975ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58085,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59611 req_id = 69c248ea-192e-4003-b05a-4a8cdf65fc6e
39752025-04-30T06:43:01.975ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
39762025-04-30T06:43:01.975ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58085, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59611 req_id = 69c248ea-192e-4003-b05a-4a8cdf65fc6e
39772025-04-30T06:43:01.975ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
39782025-04-30T06:43:01.975ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
39792025-04-30T06:43:01.975ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
39802025-04-30T06:43:01.980ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7fab83e-f53a-4f68-a475-0ae08aca1be1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39812025-04-30T06:43:01.980ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
39822025-04-30T06:43:01.992ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d6c0ef1-1bd3-420b-9b2b-e43c25c2a8d5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39832025-04-30T06:43:01.993ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
39842025-04-30T06:43:02.021ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
39852025-04-30T06:43:02.032ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46af5c05-d5b4-408a-a064-443665caf1a6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39862025-04-30T06:43:02.032ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
39872025-04-30T06:43:02.035ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62a70a08-0120-4bb8-8df1-c7381c344e6e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39882025-04-30T06:43:02.035ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
39892025-04-30T06:43:02.037ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7445cd88-b78c-4410-9a1d-1680f6df7c1b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39902025-04-30T06:43:02.037ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
39912025-04-30T06:43:02.039ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b76c182-294f-4882-bef9-44e565f3367a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39922025-04-30T06:43:02.039ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
39932025-04-30T06:43:02.041ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5eb81fe-0f6b-479b-bf66-ae4712dd312a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39942025-04-30T06:43:02.041ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
39952025-04-30T06:43:02.043ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b0847d1-028e-4a21-806a-bf2961f75e93", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39962025-04-30T06:43:02.043ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
39972025-04-30T06:43:02.044ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46d928bd-0281-4325-8af5-7a4bf994273b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
39982025-04-30T06:43:02.044ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
39992025-04-30T06:43:02.045ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17f4a8e9-f865-4ca2-93d7-68acd9967171", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40002025-04-30T06:43:02.045ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
40012025-04-30T06:43:02.046ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc1df672-3e94-4627-8e9b-d9199de6a534", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40022025-04-30T06:43:02.046ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
40032025-04-30T06:43:02.047ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bafa8f8-d041-473a-b89e-fbcef9243bb9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40042025-04-30T06:43:02.047ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
40052025-04-30T06:43:02.049ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1626fa6-412d-453f-b31a-cc7405730cb9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40062025-04-30T06:43:02.049ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
40072025-04-30T06:43:02.052ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1295b0b-02f3-4452-b686-e0ca02847a7b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40082025-04-30T06:43:02.052ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
40092025-04-30T06:43:02.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bd32e23-f568-40ff-9241-8a9af6b2dfa3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40102025-04-30T06:43:02.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
40112025-04-30T06:43:02.058ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7abf5a6-ce1c-46b4-9539-d5bfa46d455d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40122025-04-30T06:43:02.058ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
40132025-04-30T06:43:02.061ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e33d076-5b76-4dfc-9fd7-b601354ca153", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40142025-04-30T06:43:02.061ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
40152025-04-30T06:43:02.065ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ad1c8f8-73b8-46e0-9f70-87567d511c7c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40162025-04-30T06:43:02.071ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
40172025-04-30T06:43:02.072ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40182025-04-30T06:43:02.072ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40192025-04-30T06:43:02.072ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40202025-04-30T06:43:02.072ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
40212025-04-30T06:43:02.073ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40222025-04-30T06:43:02.075ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce6f591a-195c-4265-be99-c417ef213e75", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40232025-04-30T06:43:02.076ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
40242025-04-30T06:43:02.076ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14670da9-d27a-4ede-b2b4-73e789653b67", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40252025-04-30T06:43:02.077ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
40262025-04-30T06:43:02.077ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40272025-04-30T06:43:02.082ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40282025-04-30T06:43:02.082ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40292025-04-30T06:43:02.082ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
40302025-04-30T06:43:02.082ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40312025-04-30T06:43:02.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1c65f1d-fbab-4766-b3a3-c855d4334b76", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40322025-04-30T06:43:02.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
40332025-04-30T06:43:02.086ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40342025-04-30T06:43:02.086ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
40352025-04-30T06:43:02.086ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40362025-04-30T06:43:02.087ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5592394e-07d3-4256-b52f-8d3f10be0537", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40372025-04-30T06:43:02.087ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
40382025-04-30T06:43:02.089ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a55615b4-e84f-4a8b-acce-406235b77b8b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40392025-04-30T06:43:02.089ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
40402025-04-30T06:43:02.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5737ad8b-e7ed-48b6-b76d-a2300243986c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40412025-04-30T06:43:02.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
40422025-04-30T06:43:02.094ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93ddb823-3c01-4720-8f6a-9833d2e3c166", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40432025-04-30T06:43:02.094ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
40442025-04-30T06:43:02.097ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48bcdd0f-eae6-422c-8c31-0e12491528c6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40452025-04-30T06:43:02.097ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
40462025-04-30T06:43:02.099ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4f9024d-e2ee-4eea-b47c-d943fa37eba9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40472025-04-30T06:43:02.099ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
40482025-04-30T06:43:02.101ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b0e5717-7fdb-430c-a68f-7899ec81ea75", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40492025-04-30T06:43:02.101ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
40502025-04-30T06:43:02.103ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "095eed65-3956-4527-8277-778a1d92fa44", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40512025-04-30T06:43:02.103ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
40522025-04-30T06:43:02.105ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed272f4d-5366-4fb4-ae94-354963b2c6da", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40532025-04-30T06:43:02.105ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
40542025-04-30T06:43:02.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d4ef4e9-6231-4496-be1f-a4cc8077c6e2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40552025-04-30T06:43:02.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
40562025-04-30T06:43:02.107ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48f56529-78a8-4c8c-be97-cfd6328641c8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40572025-04-30T06:43:02.108ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
40582025-04-30T06:43:02.109ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37897fd7-d449-4faa-9345-b7062c681988", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40592025-04-30T06:43:02.109ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
40602025-04-30T06:43:02.111ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66aca9b9-da25-4200-b15a-fb25629ca310", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40612025-04-30T06:43:02.111ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
40622025-04-30T06:43:02.112ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb9e7717-f7ec-4bb7-aa6d-13d54c83f029", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40632025-04-30T06:43:02.112ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
40642025-04-30T06:43:02.114ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05fd7b71-943d-4345-a385-26def04eed2d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40652025-04-30T06:43:02.115ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
40662025-04-30T06:43:02.119ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ab97f06-eeaa-4180-ae70-45035a1a4e10", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40672025-04-30T06:43:02.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
40682025-04-30T06:43:02.122ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
40692025-04-30T06:43:02.124ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09638348-531b-4812-afc7-bb5412b4ad31", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40702025-04-30T06:43:02.125ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
40712025-04-30T06:43:02.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "140f76d9-4dc4-420e-a68a-16c13257c0a2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40722025-04-30T06:43:02.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
40732025-04-30T06:43:02.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de31e9aa-8da7-466c-87b7-e392b6e4073e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40742025-04-30T06:43:02.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
40752025-04-30T06:43:02.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f475f3e-e04e-4dbe-aa13-057592052991", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40762025-04-30T06:43:02.133ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
40772025-04-30T06:43:02.134ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e68738db-eb15-41b2-b08a-f2383d3ab32d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40782025-04-30T06:43:02.135ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
40792025-04-30T06:43:02.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4465a94-57b6-408b-bf90-7993f5852430", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40802025-04-30T06:43:02.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
40812025-04-30T06:43:02.138ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0e1bf32-46e1-4172-aee1-16eb30bc207a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40822025-04-30T06:43:02.138ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
40832025-04-30T06:43:02.139ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3b7604c-b132-4cc1-b9a1-9fa1624c712f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
40842025-04-30T06:43:02.139ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40852025-04-30T06:43:02.139ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40862025-04-30T06:43:02.139ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40872025-04-30T06:43:02.139ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40882025-04-30T06:43:02.140ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40892025-04-30T06:43:02.142ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40902025-04-30T06:43:02.147ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40912025-04-30T06:43:02.147ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40922025-04-30T06:43:02.147ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40932025-04-30T06:43:02.147ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40942025-04-30T06:43:02.147ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40952025-04-30T06:43:02.147ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40962025-04-30T06:43:02.150ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40972025-04-30T06:43:02.154ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40982025-04-30T06:43:02.154ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40992025-04-30T06:43:02.154ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41002025-04-30T06:43:02.155ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41012025-04-30T06:43:02.246ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
41022025-04-30T06:43:02.247ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
41032025-04-30T06:43:02.247ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41042025-04-30T06:43:02.249ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41052025-04-30T06:43:02.249ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41062025-04-30T06:43:02.249ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
41072025-04-30T06:43:02.250ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41082025-04-30T06:43:02.251ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:37066 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
41092025-04-30T06:43:02.252ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:37066 req_id = 47dd5039-fadc-48dd-a335-bdfc837653fa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
41102025-04-30T06:43:02.252ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 667 local_addr = [::1]:51119 method = GET remote_addr = [::1]:37066 req_id = 47dd5039-fadc-48dd-a335-bdfc837653fa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
41112025-04-30T06:43:02.253ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "47dd5039-fadc-48dd-a335-bdfc837653fa", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
41122025-04-30T06:43:02.255ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
41132025-04-30T06:43:02.255ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
41142025-04-30T06:43:02.255ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
41152025-04-30T06:43:02.263ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
41162025-04-30T06:43:02.263ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41172025-04-30T06:43:02.263ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41182025-04-30T06:43:02.263ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41192025-04-30T06:43:02.264ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41202025-04-30T06:43:02.264ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41212025-04-30T06:43:02.270ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41222025-04-30T06:43:02.270ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41232025-04-30T06:43:02.271ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
41242025-04-30T06:43:02.271ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41252025-04-30T06:43:02.274ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41262025-04-30T06:43:02.278ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
41272025-04-30T06:43:02.278ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41282025-04-30T06:43:02.279ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41292025-04-30T06:43:02.279ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41302025-04-30T06:43:02.283ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41312025-04-30T06:43:02.283ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
41322025-04-30T06:43:02.284ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41332025-04-30T06:43:02.356ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
41342025-04-30T06:43:02.357ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
41352025-04-30T06:43:02.357ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 508.216699ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
41362025-04-30T06:43:02.357ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
41372025-04-30T06:43:02.357ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
41382025-04-30T06:43:02.357ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41392025-04-30T06:43:02.357ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41402025-04-30T06:43:02.358ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41412025-04-30T06:43:02.358ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41422025-04-30T06:43:02.369ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41432025-04-30T06:43:02.383ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
41442025-04-30T06:43:02.390ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41452025-04-30T06:43:02.390ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41462025-04-30T06:43:02.391ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41472025-04-30T06:43:02.396ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
41482025-04-30T06:43:02.424ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
41492025-04-30T06:43:02.424ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
41502025-04-30T06:43:02.425ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bf 30 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .0..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63598 req_id = c9e9b412-9a7d-414a-b0e8-e90d00bfebc8
41512025-04-30T06:43:02.425ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48944,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63598 req_id = c9e9b412-9a7d-414a-b0e8-e90d00bfebc8
41522025-04-30T06:43:02.425ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
41532025-04-30T06:43:02.425ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
41542025-04-30T06:43:02.425ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63598 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c9e9b412-9a7d-414a-b0e8-e90d00bfebc8
41552025-04-30T06:43:02.426ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4247, tv_nsec: 224939176 } })
41562025-04-30T06:43:02.426ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
41572025-04-30T06:43:02.426ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 34 93 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 4...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63101 req_id = 331f4bec-9bad-4be8-8b6c-c8790d9f6f66
41582025-04-30T06:43:02.426ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13459,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63101 req_id = 331f4bec-9bad-4be8-8b6c-c8790d9f6f66
41592025-04-30T06:43:02.426ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
41602025-04-30T06:43:02.426ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63101 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 331f4bec-9bad-4be8-8b6c-c8790d9f6f66
41612025-04-30T06:43:02.581ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
41622025-04-30T06:43:02.582ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41632025-04-30T06:43:02.582ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41642025-04-30T06:43:02.582ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41652025-04-30T06:43:02.582ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
41662025-04-30T06:43:02.582ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41672025-04-30T06:43:02.582ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1e 86 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51043 req_id = df6a6e68-6a02-4ec0-99f4-3f03227cbf89
41682025-04-30T06:43:02.583ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7814,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51043 req_id = df6a6e68-6a02-4ec0-99f4-3f03227cbf89
41692025-04-30T06:43:02.583ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
41702025-04-30T06:43:02.583ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7814, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51043 req_id = df6a6e68-6a02-4ec0-99f4-3f03227cbf89
41712025-04-30T06:43:02.583ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
41722025-04-30T06:43:02.583ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
41732025-04-30T06:43:02.583ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
41742025-04-30T06:43:02.585ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e710c9c-6f4d-4104-9f7d-0a24ba823ee7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
41752025-04-30T06:43:02.585ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
41762025-04-30T06:43:02.587ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b3243d9-2f0c-4212-bd71-3b66c9f86e5c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
41772025-04-30T06:43:02.587ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
41782025-04-30T06:43:02.593ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41792025-04-30T06:43:02.598ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41802025-04-30T06:43:02.598ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41812025-04-30T06:43:02.598ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
41822025-04-30T06:43:02.599ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41832025-04-30T06:43:02.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0bcd6cd-6f51-4ad9-a680-674986670759", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
41842025-04-30T06:43:02.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
41852025-04-30T06:43:02.602ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41862025-04-30T06:43:02.602ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
41872025-04-30T06:43:02.603ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41882025-04-30T06:43:02.605ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c498781-2a75-4bf9-83ea-966079b56b5d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:01 GMT"} })
41892025-04-30T06:43:02.605ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
41902025-04-30T06:43:02.618ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e3d7758-93a9-4d6a-a4b8-94a58c242aa4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
41912025-04-30T06:43:02.619ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
41922025-04-30T06:43:02.632ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cf69862-997a-4281-a20e-049acc908416", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
41932025-04-30T06:43:02.632ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
41942025-04-30T06:43:02.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd83f670-7644-4c2b-b339-517cfe18ba88", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
41952025-04-30T06:43:02.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
41962025-04-30T06:43:02.683ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41972025-04-30T06:43:02.683ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41982025-04-30T06:43:02.684ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
41992025-04-30T06:43:02.685ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
42002025-04-30T06:43:02.687ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bf81739-aa58-4705-b83b-4b8caa504eb4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42012025-04-30T06:43:02.687ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
42022025-04-30T06:43:02.690ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2d4c788-1257-44a7-ac10-de0a2e1674b7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42032025-04-30T06:43:02.690ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
42042025-04-30T06:43:02.691ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
42052025-04-30T06:43:02.691ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
42062025-04-30T06:43:02.697ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
42072025-04-30T06:43:02.698ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
42082025-04-30T06:43:02.698ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42092025-04-30T06:43:02.699ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "610fe6e9-5261-48ef-822b-3d8038ac120e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42102025-04-30T06:43:02.699ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
42112025-04-30T06:43:02.701ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8a8dfeb-7308-4368-8a76-63c3b0c7eb2b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42122025-04-30T06:43:02.701ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
42132025-04-30T06:43:02.702ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15784823-2d7e-4300-b0ae-944ec7a62f08", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42142025-04-30T06:43:02.702ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
42152025-04-30T06:43:02.732ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42162025-04-30T06:43:02.732ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a9b6776-389b-4a00-a3e7-87bacfd0edde", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42172025-04-30T06:43:02.733ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
42182025-04-30T06:43:02.733ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42192025-04-30T06:43:02.750ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42202025-04-30T06:43:02.750ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42212025-04-30T06:43:02.750ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42222025-04-30T06:43:02.750ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
42232025-04-30T06:43:02.751ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42242025-04-30T06:43:02.752ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42252025-04-30T06:43:02.753ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42262025-04-30T06:43:02.753ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42272025-04-30T06:43:02.753ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42282025-04-30T06:43:02.756ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ff0d932-9998-402b-b677-726484cbb515", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42292025-04-30T06:43:02.756ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
42302025-04-30T06:43:02.756ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42312025-04-30T06:43:02.770ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42322025-04-30T06:43:02.772ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42332025-04-30T06:43:02.772ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
42342025-04-30T06:43:02.772ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42352025-04-30T06:43:02.777ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42362025-04-30T06:43:02.777ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
42372025-04-30T06:43:02.777ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42382025-04-30T06:43:02.782ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b088592-c35b-4531-bee3-29130273c81a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42392025-04-30T06:43:02.783ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
42402025-04-30T06:43:02.785ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
42412025-04-30T06:43:02.789ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "657beae1-a1c6-4505-93c5-dd4c767b28a6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42422025-04-30T06:43:02.789ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
42432025-04-30T06:43:02.791ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1493c3b2-3b50-4c65-b4fb-58072c4fd587", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42442025-04-30T06:43:02.791ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
42452025-04-30T06:43:02.793ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "291a2ec7-7931-4cbc-b921-a4905f85276f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42462025-04-30T06:43:02.794ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
42472025-04-30T06:43:02.798ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "087ba88e-439d-4fda-b597-4f144175737f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42482025-04-30T06:43:02.798ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
42492025-04-30T06:43:02.799ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7232bd50-9b3a-44bd-81e4-291829287517", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42502025-04-30T06:43:02.799ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
42512025-04-30T06:43:02.800ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9a6d8d3-4fb6-49ab-90fb-2e1d1d1489b6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42522025-04-30T06:43:02.801ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
42532025-04-30T06:43:02.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c01157b-265e-448d-b934-1e7eec32bbc0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42542025-04-30T06:43:02.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
42552025-04-30T06:43:02.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "896e3bc6-0341-4ab6-a005-bbf43f82806a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42562025-04-30T06:43:02.804ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
42572025-04-30T06:43:02.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5790e4a1-c262-4dbf-a17d-9707c936c9f8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42582025-04-30T06:43:02.807ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
42592025-04-30T06:43:02.810ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99604091-4740-46f8-b249-604c1eeb8961", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42602025-04-30T06:43:02.810ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
42612025-04-30T06:43:02.817ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
42622025-04-30T06:43:02.819ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "547b78de-07ff-4b31-bf43-82f5a036b6c3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42632025-04-30T06:43:02.820ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
42642025-04-30T06:43:02.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cea186a-9f99-400c-b23b-c4003c20c85f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42652025-04-30T06:43:02.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
42662025-04-30T06:43:02.835ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0edb7c6c-88a6-4a1b-9f59-032442f729a9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42672025-04-30T06:43:02.835ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
42682025-04-30T06:43:02.837ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a318934-a68f-40ff-bf3c-1ff437b5af4c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42692025-04-30T06:43:02.837ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
42702025-04-30T06:43:02.839ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fef0577-6930-44a4-8eee-f889e28420af", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42712025-04-30T06:43:02.839ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
42722025-04-30T06:43:02.840ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54f3bd48-b2ea-4029-a03a-f84db752d0ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42732025-04-30T06:43:02.840ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
42742025-04-30T06:43:02.841ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3238b46-98a8-4533-95e5-bfb5f1b93ab3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42752025-04-30T06:43:02.841ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
42762025-04-30T06:43:02.848ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1f6f928-88ef-4899-a428-f09065957118", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42772025-04-30T06:43:02.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
42782025-04-30T06:43:02.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cbf0470-79f6-4288-a24a-f92a84903db0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42792025-04-30T06:43:02.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
42802025-04-30T06:43:02.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23bbfead-121c-44d0-a6dc-2b1abdc77bb4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42812025-04-30T06:43:02.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
42822025-04-30T06:43:02.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa9d1e33-5224-4456-8c84-e856f4031d45", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42832025-04-30T06:43:02.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
42842025-04-30T06:43:02.850ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cecc0cb4-578f-4746-9bb6-80bac87de6fd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42852025-04-30T06:43:02.850ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
42862025-04-30T06:43:02.854ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2834ea6a-b231-48bf-9e14-ac5a5186c0e3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42872025-04-30T06:43:02.855ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
42882025-04-30T06:43:02.858ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84498ac6-e9a9-4517-905c-8ac63c506d19", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42892025-04-30T06:43:02.858ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
42902025-04-30T06:43:02.861ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf596cd3-1b10-4078-b1ac-dd365c1c13af", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42912025-04-30T06:43:02.862ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
42922025-04-30T06:43:02.865ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ac2bc71-cfce-4b48-b243-82185bb411f6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42932025-04-30T06:43:02.865ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
42942025-04-30T06:43:02.866ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf3d9fff-f7a7-4e92-873e-0550a518340c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
42952025-04-30T06:43:02.867ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
42962025-04-30T06:43:02.868ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42972025-04-30T06:43:02.868ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42982025-04-30T06:43:02.868ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42992025-04-30T06:43:02.868ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43002025-04-30T06:43:02.869ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43012025-04-30T06:43:02.870ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f9c2991-2cc5-48a3-80ff-d4b7ddfeeb82", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
43022025-04-30T06:43:02.870ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
43032025-04-30T06:43:02.872ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffd533be-f4b1-4e4a-a536-d46e96fd59db", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
43042025-04-30T06:43:02.872ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
43052025-04-30T06:43:02.873ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43062025-04-30T06:43:02.880ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43072025-04-30T06:43:02.880ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43082025-04-30T06:43:02.880ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43092025-04-30T06:43:02.881ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43102025-04-30T06:43:02.888ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6693cf35-cc49-45a4-b737-7e7f7fc7a80a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
43112025-04-30T06:43:02.888ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
43122025-04-30T06:43:02.889ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43132025-04-30T06:43:02.889ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43142025-04-30T06:43:02.889ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43152025-04-30T06:43:02.889ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43162025-04-30T06:43:02.889ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43172025-04-30T06:43:02.889ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43182025-04-30T06:43:02.890ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43192025-04-30T06:43:02.893ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43202025-04-30T06:43:02.903ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
43212025-04-30T06:43:02.903ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
43222025-04-30T06:43:02.903ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43232025-04-30T06:43:02.903ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43242025-04-30T06:43:02.903ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43252025-04-30T06:43:02.906ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43262025-04-30T06:43:02.924ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43272025-04-30T06:43:02.931ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
43282025-04-30T06:43:02.931ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43292025-04-30T06:43:02.932ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43302025-04-30T06:43:02.932ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43312025-04-30T06:43:03.026ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
43322025-04-30T06:43:03.026ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
43332025-04-30T06:43:03.027ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:33229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43342025-04-30T06:43:03.031ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:33229 req_id = ecd67d99-4b36-423a-babb-149556b4f6be sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
43352025-04-30T06:43:03.031ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 564 local_addr = [::1]:51119 method = GET remote_addr = [::1]:33229 req_id = ecd67d99-4b36-423a-babb-149556b4f6be response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
43362025-04-30T06:43:03.033ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd67d99-4b36-423a-babb-149556b4f6be", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
43372025-04-30T06:43:03.044ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
43382025-04-30T06:43:03.044ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
43392025-04-30T06:43:03.044ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
43402025-04-30T06:43:03.048ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
43412025-04-30T06:43:03.048ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43422025-04-30T06:43:03.048ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43432025-04-30T06:43:03.048ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43442025-04-30T06:43:03.049ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43452025-04-30T06:43:03.057ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43462025-04-30T06:43:03.061ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
43472025-04-30T06:43:03.061ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43482025-04-30T06:43:03.061ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43492025-04-30T06:43:03.062ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43502025-04-30T06:43:03.095ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
43512025-04-30T06:43:03.130ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
43522025-04-30T06:43:03.130ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
43532025-04-30T06:43:03.131ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 773.386532ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
43542025-04-30T06:43:03.131ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
43552025-04-30T06:43:03.131ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
43562025-04-30T06:43:03.131ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43572025-04-30T06:43:03.131ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43582025-04-30T06:43:03.131ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43592025-04-30T06:43:03.132ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43602025-04-30T06:43:03.145ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43612025-04-30T06:43:03.150ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
43622025-04-30T06:43:03.150ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43632025-04-30T06:43:03.150ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43642025-04-30T06:43:03.150ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43652025-04-30T06:43:03.151ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43662025-04-30T06:43:03.151ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43672025-04-30T06:43:03.151ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43682025-04-30T06:43:03.151ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43692025-04-30T06:43:03.151ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43702025-04-30T06:43:03.154ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
43712025-04-30T06:43:03.154ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
43722025-04-30T06:43:03.155ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8e 32 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .2..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60720 req_id = f2eba65d-131b-4ba1-977e-96d558023ae5
43732025-04-30T06:43:03.155ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36402,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60720 req_id = f2eba65d-131b-4ba1-977e-96d558023ae5
43742025-04-30T06:43:03.156ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
43752025-04-30T06:43:03.156ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
43762025-04-30T06:43:03.156ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60720 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f2eba65d-131b-4ba1-977e-96d558023ae5
43772025-04-30T06:43:03.156ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43782025-04-30T06:43:03.162ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43792025-04-30T06:43:03.163ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43802025-04-30T06:43:03.163ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43812025-04-30T06:43:03.163ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43822025-04-30T06:43:03.165ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4247, tv_nsec: 963457785 } })
43832025-04-30T06:43:03.165ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
43842025-04-30T06:43:03.169ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8b 39 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .9..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58460 req_id = 99c8aa00-15db-4146-ae0b-46d53def9343
43852025-04-30T06:43:03.171ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35641,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58460 req_id = 99c8aa00-15db-4146-ae0b-46d53def9343
43862025-04-30T06:43:03.171ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
43872025-04-30T06:43:03.171ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58460 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 99c8aa00-15db-4146-ae0b-46d53def9343
43882025-04-30T06:43:03.171ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43892025-04-30T06:43:03.171ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43902025-04-30T06:43:03.172ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43912025-04-30T06:43:03.251ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
43922025-04-30T06:43:03.252ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a1 f9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56314 req_id = 426deb3a-a59d-4f22-b2ae-07d39e1690d1
43932025-04-30T06:43:03.252ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41465,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56314 req_id = 426deb3a-a59d-4f22-b2ae-07d39e1690d1
43942025-04-30T06:43:03.252ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
43952025-04-30T06:43:03.253ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41465, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56314 req_id = 426deb3a-a59d-4f22-b2ae-07d39e1690d1
43962025-04-30T06:43:03.253ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
43972025-04-30T06:43:03.253ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
43982025-04-30T06:43:03.253ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
43992025-04-30T06:43:03.255ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2544f467-8f99-4344-992e-c71ad68a21fb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44002025-04-30T06:43:03.255ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
44012025-04-30T06:43:03.257ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df1b9357-13f3-4f8c-93be-70fd09fd8868", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44022025-04-30T06:43:03.257ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
44032025-04-30T06:43:03.271ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "925aadcb-4996-402b-90d1-2af93af10a93", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44042025-04-30T06:43:03.271ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
44052025-04-30T06:43:03.279ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40304aec-08b1-4998-ae1f-0a36e0b533fd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44062025-04-30T06:43:03.279ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
44072025-04-30T06:43:03.282ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66f8f772-c531-48d6-9aa7-76db3fdefcdc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44082025-04-30T06:43:03.282ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
44092025-04-30T06:43:03.284ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0effc579-1c35-4612-bb42-1845520cc0d8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44102025-04-30T06:43:03.284ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
44112025-04-30T06:43:03.286ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a377c74d-75cd-4c65-948a-c33e43eb5d64", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44122025-04-30T06:43:03.286ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
44132025-04-30T06:43:03.288ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3d419fc-2f90-4562-8aa9-7c2fa9da8a5d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44142025-04-30T06:43:03.288ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
44152025-04-30T06:43:03.289ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc0f43f4-6cd2-434e-ac90-7759bbe9c045", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44162025-04-30T06:43:03.289ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
44172025-04-30T06:43:03.290ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "863d1bba-e745-46ce-a349-c13fce857331", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44182025-04-30T06:43:03.290ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
44192025-04-30T06:43:03.292ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fa86a0e-8294-4028-9bcb-ee61b46a798d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44202025-04-30T06:43:03.293ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
44212025-04-30T06:43:03.293ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
44222025-04-30T06:43:03.298ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dfcfaa9-00c1-4dc9-95a1-b09bdeaa0d95", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44232025-04-30T06:43:03.299ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
44242025-04-30T06:43:03.300ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b985876-3812-47aa-8c69-f5a75bfb792b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44252025-04-30T06:43:03.301ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
44262025-04-30T06:43:03.303ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1f30b9f-7b62-4757-a88f-72dde5700d5c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44272025-04-30T06:43:03.303ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
44282025-04-30T06:43:03.307ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b22600d-aef0-4947-aaf1-314ebd53ab00", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44292025-04-30T06:43:03.308ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
44302025-04-30T06:43:03.309ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "916f7c18-2127-4c8b-ad5d-9bc9ae21fb00", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44312025-04-30T06:43:03.310ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
44322025-04-30T06:43:03.312ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "febd30bf-6f48-43fa-967e-613b588cac38", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44332025-04-30T06:43:03.312ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
44342025-04-30T06:43:03.314ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ca6382d-a687-4c22-a0dc-b6f04b358324", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44352025-04-30T06:43:03.314ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
44362025-04-30T06:43:03.316ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "808cb354-0c41-4063-9a8f-90167ec9d8ac", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44372025-04-30T06:43:03.316ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
44382025-04-30T06:43:03.318ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "688eb236-4a40-4861-a1ed-b769f2fea938", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44392025-04-30T06:43:03.318ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
44402025-04-30T06:43:03.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d90ae64b-ca68-4974-8a9e-487dab44f3b4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44412025-04-30T06:43:03.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
44422025-04-30T06:43:03.320ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5ae4d84-59ff-4a47-80ce-afc235d6f399", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44432025-04-30T06:43:03.320ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
44442025-04-30T06:43:03.321ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01e3007a-9238-4288-b0b3-3f583d4731b1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44452025-04-30T06:43:03.322ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
44462025-04-30T06:43:03.323ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "662758f9-67b2-48ff-b587-d2a501b6665a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44472025-04-30T06:43:03.323ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
44482025-04-30T06:43:03.325ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0aa1906d-c960-45eb-bc08-07ba87d41235", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44492025-04-30T06:43:03.326ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
44502025-04-30T06:43:03.327ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd587c20-5d0e-441b-999c-a375c9b6d3ec", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44512025-04-30T06:43:03.327ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
44522025-04-30T06:43:03.329ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b45d1ac3-f24d-4d14-aa9c-2d1ebfa9f135", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44532025-04-30T06:43:03.329ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
44542025-04-30T06:43:03.331ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ace46db-1a81-4721-90e8-97d050df856c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44552025-04-30T06:43:03.331ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
44562025-04-30T06:43:03.339ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee57e0b6-339c-4625-bf19-46982b41af03", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44572025-04-30T06:43:03.339ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
44582025-04-30T06:43:03.341ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b6037bd-d622-4f04-aee0-7c914dc54bf8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44592025-04-30T06:43:03.341ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
44602025-04-30T06:43:03.342ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8836c837-90e5-41fd-ac92-1e2b5e5d1b9f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44612025-04-30T06:43:03.342ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
44622025-04-30T06:43:03.343ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14b44877-1d55-4533-b95f-96b1f973d8ee", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44632025-04-30T06:43:03.343ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
44642025-04-30T06:43:03.344ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44652025-04-30T06:43:03.344ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44662025-04-30T06:43:03.344ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44672025-04-30T06:43:03.344ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
44682025-04-30T06:43:03.345ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44692025-04-30T06:43:03.345ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da8255c2-ce3b-403e-927f-b0e99ae8cb09", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44702025-04-30T06:43:03.345ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
44712025-04-30T06:43:03.346ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2339e3a-623d-44b2-9c8d-3901ab22dcc0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44722025-04-30T06:43:03.346ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
44732025-04-30T06:43:03.347ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b0d9e81-5c69-44f3-a303-890db9b8892f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44742025-04-30T06:43:03.347ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
44752025-04-30T06:43:03.348ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44762025-04-30T06:43:03.352ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44772025-04-30T06:43:03.352ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44782025-04-30T06:43:03.353ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
44792025-04-30T06:43:03.353ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44802025-04-30T06:43:03.356ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44812025-04-30T06:43:03.356ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
44822025-04-30T06:43:03.356ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44832025-04-30T06:43:03.357ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4c26ab5-434e-4bd4-8f06-65aae3492f53", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44842025-04-30T06:43:03.358ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
44852025-04-30T06:43:03.360ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d33bc919-4bbc-493b-aaad-3e5c6ca189dc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44862025-04-30T06:43:03.360ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
44872025-04-30T06:43:03.363ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3dc3164-c815-4cd0-b47f-5919469563c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44882025-04-30T06:43:03.364ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
44892025-04-30T06:43:03.366ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bad23a0-f6c8-46cb-8a96-415e7963ff72", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44902025-04-30T06:43:03.367ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
44912025-04-30T06:43:03.369ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb874cfd-a5e1-4f3e-bcba-f03304d4fc1f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44922025-04-30T06:43:03.371ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
44932025-04-30T06:43:03.377ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ba11cf0-d70c-400d-892e-739ccda0a488", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44942025-04-30T06:43:03.377ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
44952025-04-30T06:43:03.378ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3539c765-459a-4cb4-83e1-8e3bc96df0ed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44962025-04-30T06:43:03.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
44972025-04-30T06:43:03.386ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6eae5ed-c086-487d-9fc6-c5deb2f6a02b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
44982025-04-30T06:43:03.386ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
44992025-04-30T06:43:03.389ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2d74c08-cfdd-4195-a6da-4f836ec4fa10", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
45002025-04-30T06:43:03.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
45012025-04-30T06:43:03.396ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
45022025-04-30T06:43:03.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "328edba3-bd3f-4b87-b296-f5ba1de3c061", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
45032025-04-30T06:43:03.396ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
45042025-04-30T06:43:03.397ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45052025-04-30T06:43:03.397ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45062025-04-30T06:43:03.397ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45072025-04-30T06:43:03.398ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45082025-04-30T06:43:03.402ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45092025-04-30T06:43:03.407ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
45102025-04-30T06:43:03.407ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
45112025-04-30T06:43:03.407ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45122025-04-30T06:43:03.407ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45132025-04-30T06:43:03.407ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45142025-04-30T06:43:03.408ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45152025-04-30T06:43:03.420ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45162025-04-30T06:43:03.423ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
45172025-04-30T06:43:03.423ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45182025-04-30T06:43:03.423ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45192025-04-30T06:43:03.424ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45202025-04-30T06:43:03.496ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
45212025-04-30T06:43:03.496ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
45222025-04-30T06:43:03.497ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45232025-04-30T06:43:03.497ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45242025-04-30T06:43:03.497ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45252025-04-30T06:43:03.497ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
45262025-04-30T06:43:03.500ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45272025-04-30T06:43:03.500ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:53004 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
45282025-04-30T06:43:03.501ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:53004 req_id = 127d4222-76a5-4f11-a16f-7803b0f44127 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
45292025-04-30T06:43:03.501ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 705 local_addr = [::1]:51119 method = GET remote_addr = [::1]:53004 req_id = 127d4222-76a5-4f11-a16f-7803b0f44127 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
45302025-04-30T06:43:03.502ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "127d4222-76a5-4f11-a16f-7803b0f44127", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
45312025-04-30T06:43:03.508ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
45322025-04-30T06:43:03.508ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
45332025-04-30T06:43:03.508ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
45342025-04-30T06:43:03.512ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
45352025-04-30T06:43:03.512ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45362025-04-30T06:43:03.512ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45372025-04-30T06:43:03.512ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45382025-04-30T06:43:03.514ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45392025-04-30T06:43:03.514ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45402025-04-30T06:43:03.523ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45412025-04-30T06:43:03.523ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45422025-04-30T06:43:03.523ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
45432025-04-30T06:43:03.524ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45442025-04-30T06:43:03.526ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45452025-04-30T06:43:03.532ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
45462025-04-30T06:43:03.532ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45472025-04-30T06:43:03.533ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45482025-04-30T06:43:03.533ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45492025-04-30T06:43:03.536ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45502025-04-30T06:43:03.536ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
45512025-04-30T06:43:03.537ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45522025-04-30T06:43:03.568ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
45532025-04-30T06:43:03.568ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
45542025-04-30T06:43:03.568ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 437.370843ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
45552025-04-30T06:43:03.568ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
45562025-04-30T06:43:03.568ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
45572025-04-30T06:43:03.568ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45582025-04-30T06:43:03.568ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45592025-04-30T06:43:03.568ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45602025-04-30T06:43:03.569ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45612025-04-30T06:43:03.570ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
45622025-04-30T06:43:03.571ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45632025-04-30T06:43:03.575ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
45642025-04-30T06:43:03.575ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45652025-04-30T06:43:03.575ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45662025-04-30T06:43:03.576ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45672025-04-30T06:43:03.590ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
45682025-04-30T06:43:03.590ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45692025-04-30T06:43:03.590ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3f438a5c-c564-4ba8-bc69-2459a5cca7cf uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45702025-04-30T06:43:03.591ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3f438a5c-c564-4ba8-bc69-2459a5cca7cf resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45712025-04-30T06:43:03.591ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3f438a5c-c564-4ba8-bc69-2459a5cca7cf roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45722025-04-30T06:43:03.592ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3f438a5c-c564-4ba8-bc69-2459a5cca7cf resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45732025-04-30T06:43:03.596ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
45742025-04-30T06:43:03.596ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
45752025-04-30T06:43:03.598ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 71 41 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 qA..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61768 req_id = 9632962f-09d3-4c17-a416-b414ebbdb677
45762025-04-30T06:43:03.598ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28993,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61768 req_id = 9632962f-09d3-4c17-a416-b414ebbdb677
45772025-04-30T06:43:03.598ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
45782025-04-30T06:43:03.598ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45792025-04-30T06:43:03.598ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61768 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9632962f-09d3-4c17-a416-b414ebbdb677
45802025-04-30T06:43:03.600ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 9329 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3f438a5c-c564-4ba8-bc69-2459a5cca7cf response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45812025-04-30T06:43:03.601ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4248, tv_nsec: 399893926 } })
45822025-04-30T06:43:03.601ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
45832025-04-30T06:43:03.601ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f438a5c-c564-4ba8-bc69-2459a5cca7cf", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
45842025-04-30T06:43:03.601ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45852025-04-30T06:43:03.601ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4408a079-e5b5-41de-b957-f7114ea15f44 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45862025-04-30T06:43:03.602ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4408a079-e5b5-41de-b957-f7114ea15f44 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45872025-04-30T06:43:03.603ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4408a079-e5b5-41de-b957-f7114ea15f44 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45882025-04-30T06:43:03.604ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4408a079-e5b5-41de-b957-f7114ea15f44 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45892025-04-30T06:43:03.604ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8d 68 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .h..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60150 req_id = 1a0a1c49-b5a2-4e8a-8093-0511304f517c
45902025-04-30T06:43:03.604ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36200,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60150 req_id = 1a0a1c49-b5a2-4e8a-8093-0511304f517c
45912025-04-30T06:43:03.604ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45922025-04-30T06:43:03.604ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60150 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1a0a1c49-b5a2-4e8a-8093-0511304f517c
45932025-04-30T06:43:03.698ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
45942025-04-30T06:43:03.698ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45952025-04-30T06:43:03.698ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45962025-04-30T06:43:03.699ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45972025-04-30T06:43:03.699ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
45982025-04-30T06:43:03.699ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45992025-04-30T06:43:03.700ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46002025-04-30T06:43:03.700ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46012025-04-30T06:43:03.701ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
46022025-04-30T06:43:03.702ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
46032025-04-30T06:43:03.702ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 100605 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4408a079-e5b5-41de-b957-f7114ea15f44 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46042025-04-30T06:43:03.703ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ce a9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54786 req_id = cde5c757-1dfb-4617-80c8-0f0d7d1f1030
46052025-04-30T06:43:03.703ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52905,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54786 req_id = cde5c757-1dfb-4617-80c8-0f0d7d1f1030
46062025-04-30T06:43:03.704ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
46072025-04-30T06:43:03.704ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52905, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54786 req_id = cde5c757-1dfb-4617-80c8-0f0d7d1f1030
46082025-04-30T06:43:03.704ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "4408a079-e5b5-41de-b957-f7114ea15f44", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46092025-04-30T06:43:03.704ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
46102025-04-30T06:43:03.704ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46112025-04-30T06:43:03.705ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
46122025-04-30T06:43:03.705ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46132025-04-30T06:43:03.721ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46142025-04-30T06:43:03.721ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46152025-04-30T06:43:03.721ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
46162025-04-30T06:43:03.721ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46172025-04-30T06:43:03.722ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
46182025-04-30T06:43:03.722ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
46192025-04-30T06:43:03.722ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
46202025-04-30T06:43:03.726ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
46212025-04-30T06:43:03.727ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
46222025-04-30T06:43:03.727ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46232025-04-30T06:43:03.727ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46242025-04-30T06:43:03.730ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46252025-04-30T06:43:03.730ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
46262025-04-30T06:43:03.731ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46272025-04-30T06:43:03.731ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
46282025-04-30T06:43:03.731ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46292025-04-30T06:43:03.732ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46302025-04-30T06:43:03.732ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46312025-04-30T06:43:03.737ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46322025-04-30T06:43:03.737ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46332025-04-30T06:43:03.739ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e691f42c-32f5-420e-8565-bfa8b51ef1a9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46342025-04-30T06:43:03.739ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
46352025-04-30T06:43:03.741ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f0a5f3-75b3-4a7c-bb06-38a9c752dffe", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46362025-04-30T06:43:03.741ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
46372025-04-30T06:43:03.745ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "729cd92e-3726-4682-a5d7-2e4bc9a2d7db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46382025-04-30T06:43:03.745ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
46392025-04-30T06:43:03.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a068e03-ce5b-4946-bd99-5dd0275ff880", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46402025-04-30T06:43:03.751ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
46412025-04-30T06:43:03.758ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d36c90e9-538b-432d-8eb4-5153dbe6a752", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46422025-04-30T06:43:03.759ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
46432025-04-30T06:43:03.763ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "536750b6-da6e-4c08-99f8-75a9fee35afd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46442025-04-30T06:43:03.763ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
46452025-04-30T06:43:03.775ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9f1ad56-d407-4a02-9ee4-c3873d45931e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46462025-04-30T06:43:03.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
46472025-04-30T06:43:03.786ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
46482025-04-30T06:43:03.788ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcb6e8bc-8a99-47e0-8880-306e32d746e8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46492025-04-30T06:43:03.788ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
46502025-04-30T06:43:03.790ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "006d53e1-2920-47d6-af7d-77b5066bb5a3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46512025-04-30T06:43:03.790ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
46522025-04-30T06:43:03.793ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84666996-e29f-4f87-a50a-58ce9abbafba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46532025-04-30T06:43:03.793ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
46542025-04-30T06:43:03.795ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de7f3265-2485-4aab-b558-bf586879a54f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46552025-04-30T06:43:03.795ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
46562025-04-30T06:43:03.796ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31c63949-73e9-4aaa-985a-84705a7262f5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46572025-04-30T06:43:03.797ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
46582025-04-30T06:43:03.799ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd69496a-09ea-44e3-bdbb-964481cbcad5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46592025-04-30T06:43:03.799ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
46602025-04-30T06:43:03.801ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
46612025-04-30T06:43:03.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27b727ff-80ca-4855-9fda-aaae8ae6935d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46622025-04-30T06:43:03.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
46632025-04-30T06:43:03.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f079a3f-9ef2-4095-ac5f-ca0b1f486f31", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46642025-04-30T06:43:03.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
46652025-04-30T06:43:03.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06cb38f1-3ff2-4622-8cce-7fdbdb649c9c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46662025-04-30T06:43:03.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
46672025-04-30T06:43:03.811ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c247ffb-ca8f-4215-a7b3-b5e7fc776d07", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46682025-04-30T06:43:03.811ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
46692025-04-30T06:43:03.814ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a1c5f7-50e3-4459-81cc-ec28d36d527f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46702025-04-30T06:43:03.814ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
46712025-04-30T06:43:03.817ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e48d8eb-7446-4562-8a24-9503e0448a30", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46722025-04-30T06:43:03.817ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
46732025-04-30T06:43:03.818ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f47b540-ecac-497a-bd6d-fa7553a15f22", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46742025-04-30T06:43:03.818ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
46752025-04-30T06:43:03.819ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13eec0c6-4f25-4ebf-b125-9731501102e4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46762025-04-30T06:43:03.819ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
46772025-04-30T06:43:03.820ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c9b8537-15ee-44e7-9055-94837f9fdc44", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46782025-04-30T06:43:03.821ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
46792025-04-30T06:43:03.822ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "954a6020-8390-4eac-9afb-6f342ad63478", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46802025-04-30T06:43:03.822ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
46812025-04-30T06:43:03.824ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "def8c789-719e-41f6-814f-e1c29ecd8cc4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46822025-04-30T06:43:03.824ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
46832025-04-30T06:43:03.826ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db0b4304-3600-4b9d-b4b3-6f5776bbdb84", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46842025-04-30T06:43:03.826ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
46852025-04-30T06:43:03.872ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46862025-04-30T06:43:03.873ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46872025-04-30T06:43:03.873ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46882025-04-30T06:43:03.873ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
46892025-04-30T06:43:03.873ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46902025-04-30T06:43:03.874ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0010418-5281-4ac5-b47e-4e181b91ab05", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
46912025-04-30T06:43:03.874ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
46922025-04-30T06:43:03.882ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46932025-04-30T06:43:03.889ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46942025-04-30T06:43:03.889ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46952025-04-30T06:43:03.889ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
46962025-04-30T06:43:03.889ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46972025-04-30T06:43:03.904ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46982025-04-30T06:43:03.904ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
46992025-04-30T06:43:03.905ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47002025-04-30T06:43:03.906ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7355d38d-e7c3-4290-ab67-e594d50dc6ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47012025-04-30T06:43:03.906ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
47022025-04-30T06:43:03.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "022a8e50-aef5-495b-a2af-5e20b5715437", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47032025-04-30T06:43:03.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
47042025-04-30T06:43:03.910ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4064564-e2a9-4c63-9833-7fd77a6d5a64", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47052025-04-30T06:43:03.910ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
47062025-04-30T06:43:03.913ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25824090-ac44-474a-89cf-a19ce632e7ea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47072025-04-30T06:43:03.913ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
47082025-04-30T06:43:03.914ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "480b2727-29cc-485a-ac8e-d0a66eaa444d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47092025-04-30T06:43:03.915ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
47102025-04-30T06:43:03.916ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e1fba51-4e3e-4fc1-a7a2-c07e98471b66", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47112025-04-30T06:43:03.916ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
47122025-04-30T06:43:03.918ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa70ab62-3175-48fb-83db-65ee334d08cc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47132025-04-30T06:43:03.918ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
47142025-04-30T06:43:03.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1440e389-1b87-4c80-9545-33523395bcdc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47152025-04-30T06:43:03.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
47162025-04-30T06:43:03.921ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3a3841c-d801-46ed-947c-849996b60388", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47172025-04-30T06:43:03.921ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
47182025-04-30T06:43:03.926ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4266900-cdf6-438e-b140-ca005e40be65", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47192025-04-30T06:43:03.926ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
47202025-04-30T06:43:03.932ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dbce045-8916-413a-b489-a2f70fa5d5be", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47212025-04-30T06:43:03.932ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
47222025-04-30T06:43:03.936ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
47232025-04-30T06:43:03.940ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11fa65f2-0974-4d2c-aff5-4ab1cb0d242e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47242025-04-30T06:43:03.940ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
47252025-04-30T06:43:03.943ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e193f63c-52e4-464f-bf56-9fa7c560426d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47262025-04-30T06:43:03.943ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
47272025-04-30T06:43:03.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "792cdeea-0c7a-4a28-bded-0cf403bac75b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47282025-04-30T06:43:03.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
47292025-04-30T06:43:03.949ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f921a4e-fd50-4892-bf1d-706741ddcbc2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47302025-04-30T06:43:03.949ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
47312025-04-30T06:43:03.950ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d01dbadd-d135-4227-b9bb-ed5a1217fec8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47322025-04-30T06:43:03.951ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
47332025-04-30T06:43:03.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c92fc2a-b98e-4d2f-a8ab-948acd822b20", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47342025-04-30T06:43:03.963ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
47352025-04-30T06:43:03.965ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fa5ef48-6c22-423e-85c3-b3a8ec08da5a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47362025-04-30T06:43:03.966ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
47372025-04-30T06:43:03.967ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4314acd3-1ef8-43d6-b291-9630d408efa9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47382025-04-30T06:43:03.967ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
47392025-04-30T06:43:03.967ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47402025-04-30T06:43:03.967ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47412025-04-30T06:43:03.968ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47422025-04-30T06:43:03.969ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47432025-04-30T06:43:03.973ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47442025-04-30T06:43:03.978ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
47452025-04-30T06:43:03.978ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
47462025-04-30T06:43:03.978ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47472025-04-30T06:43:03.978ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47482025-04-30T06:43:03.978ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47492025-04-30T06:43:03.979ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47502025-04-30T06:43:03.982ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47512025-04-30T06:43:03.987ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
47522025-04-30T06:43:03.988ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47532025-04-30T06:43:03.988ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47542025-04-30T06:43:03.989ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47552025-04-30T06:43:03.992ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47562025-04-30T06:43:03.992ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47572025-04-30T06:43:03.992ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47582025-04-30T06:43:03.992ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
47592025-04-30T06:43:03.992ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47602025-04-30T06:43:04.055ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
47612025-04-30T06:43:04.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
47622025-04-30T06:43:04.057ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:63441 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
47632025-04-30T06:43:04.057ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:63441 req_id = c0febf71-e2da-4e09-b112-5d50fc6631a0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
47642025-04-30T06:43:04.060ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 3472 local_addr = [::1]:51119 method = GET remote_addr = [::1]:63441 req_id = c0febf71-e2da-4e09-b112-5d50fc6631a0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
47652025-04-30T06:43:04.060ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47662025-04-30T06:43:04.069ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47672025-04-30T06:43:04.069ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47682025-04-30T06:43:04.069ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
47692025-04-30T06:43:04.069ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47702025-04-30T06:43:04.072ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0febf71-e2da-4e09-b112-5d50fc6631a0", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
47712025-04-30T06:43:04.075ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
47722025-04-30T06:43:04.075ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
47732025-04-30T06:43:04.075ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
47742025-04-30T06:43:04.082ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
47752025-04-30T06:43:04.082ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47762025-04-30T06:43:04.082ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47772025-04-30T06:43:04.082ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47782025-04-30T06:43:04.083ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47792025-04-30T06:43:04.102ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47802025-04-30T06:43:04.111ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47812025-04-30T06:43:04.111ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47822025-04-30T06:43:04.111ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47832025-04-30T06:43:04.112ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47842025-04-30T06:43:04.113ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47852025-04-30T06:43:04.113ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
47862025-04-30T06:43:04.113ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47872025-04-30T06:43:04.161ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
47882025-04-30T06:43:04.162ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
47892025-04-30T06:43:04.162ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 594.023372ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
47902025-04-30T06:43:04.163ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
47912025-04-30T06:43:04.163ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
47922025-04-30T06:43:04.163ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47932025-04-30T06:43:04.163ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47942025-04-30T06:43:04.164ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47952025-04-30T06:43:04.164ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47962025-04-30T06:43:04.170ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47972025-04-30T06:43:04.176ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47982025-04-30T06:43:04.176ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47992025-04-30T06:43:04.177ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48002025-04-30T06:43:04.178ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48012025-04-30T06:43:04.184ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
48022025-04-30T06:43:04.184ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
48032025-04-30T06:43:04.185ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
48042025-04-30T06:43:04.186ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 95 9a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62810 req_id = a85b814d-6458-4425-bbf5-afe265a85eb6
48052025-04-30T06:43:04.186ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38298,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62810 req_id = a85b814d-6458-4425-bbf5-afe265a85eb6
48062025-04-30T06:43:04.186ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
48072025-04-30T06:43:04.186ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
48082025-04-30T06:43:04.186ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62810 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a85b814d-6458-4425-bbf5-afe265a85eb6
48092025-04-30T06:43:04.187ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4248, tv_nsec: 986468868 } })
48102025-04-30T06:43:04.187ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
48112025-04-30T06:43:04.188ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 55 3d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e U=..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56228 req_id = 473a7ac9-0025-4256-9683-55f6580438aa
48122025-04-30T06:43:04.188ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21821,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56228 req_id = 473a7ac9-0025-4256-9683-55f6580438aa
48132025-04-30T06:43:04.188ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
48142025-04-30T06:43:04.188ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56228 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 473a7ac9-0025-4256-9683-55f6580438aa
48152025-04-30T06:43:04.256ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
48162025-04-30T06:43:04.257ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48172025-04-30T06:43:04.257ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48182025-04-30T06:43:04.257ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48192025-04-30T06:43:04.257ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
48202025-04-30T06:43:04.258ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48212025-04-30T06:43:04.258ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b6 80 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52300 req_id = 97ce27c5-d7ac-4d3a-b24f-504be4a9035d
48222025-04-30T06:43:04.258ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46720,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52300 req_id = 97ce27c5-d7ac-4d3a-b24f-504be4a9035d
48232025-04-30T06:43:04.258ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
48242025-04-30T06:43:04.259ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46720, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52300 req_id = 97ce27c5-d7ac-4d3a-b24f-504be4a9035d
48252025-04-30T06:43:04.259ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
48262025-04-30T06:43:04.259ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
48272025-04-30T06:43:04.259ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
48282025-04-30T06:43:04.260ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48292025-04-30T06:43:04.267ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48302025-04-30T06:43:04.267ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48312025-04-30T06:43:04.267ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
48322025-04-30T06:43:04.268ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48332025-04-30T06:43:04.272ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eec26e6-f490-4620-88e5-0f41b938b9c7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48342025-04-30T06:43:04.272ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
48352025-04-30T06:43:04.273ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48362025-04-30T06:43:04.273ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
48372025-04-30T06:43:04.273ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48382025-04-30T06:43:04.275ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1699ec3-173b-46b1-b2db-26575afde18f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48392025-04-30T06:43:04.275ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
48402025-04-30T06:43:04.279ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab84113f-5d96-4554-a763-ca24314f6f94", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48412025-04-30T06:43:04.279ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
48422025-04-30T06:43:04.283ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32f5a90d-3d29-4623-81c0-196029dc54f5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48432025-04-30T06:43:04.284ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
48442025-04-30T06:43:04.288ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf3ac197-0187-4467-9ba9-d674abe861b3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48452025-04-30T06:43:04.288ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
48462025-04-30T06:43:04.291ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b33bcc3-3fce-4d90-bf4e-0b576ab0b7b2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48472025-04-30T06:43:04.291ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
48482025-04-30T06:43:04.295ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d36ae498-e857-4942-af7b-6c1930bef40d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48492025-04-30T06:43:04.301ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
48502025-04-30T06:43:04.305ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9656631f-5165-4dac-a01b-a5931a756922", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48512025-04-30T06:43:04.305ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
48522025-04-30T06:43:04.307ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04a9942b-0189-46a9-8468-9563f8659712", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48532025-04-30T06:43:04.307ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
48542025-04-30T06:43:04.307ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d067650-6ee8-4738-925f-09487632bc1b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48552025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
48562025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "658a3398-2114-40a9-99aa-7fe49109f03f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48572025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
48582025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16684f89-9aa3-4443-b9fb-7e29e0178256", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48592025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
48602025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
48612025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0b84c5c-6bf5-4268-91ad-22d5c7e918d2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48622025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
48632025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c19ee002-3be0-45b3-9124-fd24bbeee666", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48642025-04-30T06:43:04.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
48652025-04-30T06:43:04.320ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90e5496e-4715-4e66-a067-d1153fc0867e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48662025-04-30T06:43:04.320ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
48672025-04-30T06:43:04.323ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9400297a-a894-43c5-811c-bd3a3ea004e9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48682025-04-30T06:43:04.323ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
48692025-04-30T06:43:04.326ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46e1d8ec-e36b-4bab-ae51-55234c3a5430", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48702025-04-30T06:43:04.326ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
48712025-04-30T06:43:04.328ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d164f91-b278-4434-b6ec-c8daddae4c2a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48722025-04-30T06:43:04.328ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
48732025-04-30T06:43:04.331ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc0b4bf5-476a-4677-9a39-1bc9457e578e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48742025-04-30T06:43:04.331ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
48752025-04-30T06:43:04.333ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "318cec8d-9a6f-40b1-b9c5-68645488020a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48762025-04-30T06:43:04.333ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
48772025-04-30T06:43:04.335ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c4cc096-63d2-47f5-8a04-a7a82d07cbb9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48782025-04-30T06:43:04.335ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
48792025-04-30T06:43:04.336ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d007c49d-c6b4-4315-955f-8dc9fad4bda2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48802025-04-30T06:43:04.336ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
48812025-04-30T06:43:04.337ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a43ba5ba-802e-4c99-abb8-65893096dc8b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48822025-04-30T06:43:04.337ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
48832025-04-30T06:43:04.339ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b523e6fa-f0fe-4f9e-91de-73da1dc2157e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48842025-04-30T06:43:04.339ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
48852025-04-30T06:43:04.343ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98a58b9b-c9be-4d16-b1e1-7dd1e59d5db0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48862025-04-30T06:43:04.343ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
48872025-04-30T06:43:04.345ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7382dcd5-366c-4746-8029-9542566bb297", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48882025-04-30T06:43:04.345ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
48892025-04-30T06:43:04.347ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fba5e09-7448-43e6-9e9b-812a7e513f56", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48902025-04-30T06:43:04.347ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
48912025-04-30T06:43:04.350ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c66c0a07-cadd-4f80-9d60-0b4e00cddf2f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48922025-04-30T06:43:04.350ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
48932025-04-30T06:43:04.352ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7871ea95-071b-483b-ae5e-343fd9b4d81d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
48942025-04-30T06:43:04.367ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
48952025-04-30T06:43:04.367ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48962025-04-30T06:43:04.368ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48972025-04-30T06:43:04.368ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48982025-04-30T06:43:04.368ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
48992025-04-30T06:43:04.371ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49002025-04-30T06:43:04.374ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49012025-04-30T06:43:04.382ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49022025-04-30T06:43:04.382ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49032025-04-30T06:43:04.382ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49042025-04-30T06:43:04.383ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49052025-04-30T06:43:04.383ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b09f381-5d7d-4075-89c4-20877fb36cc0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49062025-04-30T06:43:04.385ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
49072025-04-30T06:43:04.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fde4123-4afc-4a0e-9981-a24f7d8bd66b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49082025-04-30T06:43:04.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
49092025-04-30T06:43:04.392ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49102025-04-30T06:43:04.392ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49112025-04-30T06:43:04.392ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49122025-04-30T06:43:04.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09d94642-f956-405f-96a2-74e7b6beede6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49132025-04-30T06:43:04.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
49142025-04-30T06:43:04.398ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71167bd2-1915-41df-b41c-759a1a63540a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49152025-04-30T06:43:04.398ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
49162025-04-30T06:43:04.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8e21ecd-c7a6-4017-b568-9688d462b9e2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49172025-04-30T06:43:04.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
49182025-04-30T06:43:04.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07e138c2-6ee6-448c-9974-9157f87fcd16", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49192025-04-30T06:43:04.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
49202025-04-30T06:43:04.402ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40ba95e2-091d-4325-9213-2fad670b5f90", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49212025-04-30T06:43:04.403ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
49222025-04-30T06:43:04.406ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3b18e60-759a-452c-b66b-2118932fbe84", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49232025-04-30T06:43:04.406ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
49242025-04-30T06:43:04.410ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11b88b95-ee68-43ea-a9ce-5d797b95e6a6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49252025-04-30T06:43:04.411ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
49262025-04-30T06:43:04.416ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59875f21-4b1c-45b5-9974-f5cd07690e9b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49272025-04-30T06:43:04.416ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
49282025-04-30T06:43:04.422ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "824804b6-fc5e-4ecd-ac83-4c3350f459e3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49292025-04-30T06:43:04.422ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
49302025-04-30T06:43:04.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f99040d0-5618-4df0-8a4f-a0af568ae5c6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49312025-04-30T06:43:04.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
49322025-04-30T06:43:04.431ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d4b29e5-8869-40f5-a68c-7b997af9787f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49332025-04-30T06:43:04.431ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
49342025-04-30T06:43:04.432ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
49352025-04-30T06:43:04.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19c24dd9-2147-440e-b1d0-a9164e63df4c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49362025-04-30T06:43:04.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
49372025-04-30T06:43:04.458ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b60f33f5-5130-4eca-81dd-f8ecad89adea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49382025-04-30T06:43:04.459ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
49392025-04-30T06:43:04.461ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34a0c25a-8af8-4610-bd8d-a6cf5cd7df36", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49402025-04-30T06:43:04.462ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
49412025-04-30T06:43:04.462ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49422025-04-30T06:43:04.462ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49432025-04-30T06:43:04.462ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49442025-04-30T06:43:04.464ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49452025-04-30T06:43:04.468ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49462025-04-30T06:43:04.475ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
49472025-04-30T06:43:04.475ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
49482025-04-30T06:43:04.475ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49492025-04-30T06:43:04.475ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49502025-04-30T06:43:04.475ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49512025-04-30T06:43:04.476ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49522025-04-30T06:43:04.478ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49532025-04-30T06:43:04.484ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
49542025-04-30T06:43:04.485ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49552025-04-30T06:43:04.485ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49562025-04-30T06:43:04.485ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49572025-04-30T06:43:04.486ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49582025-04-30T06:43:04.486ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49592025-04-30T06:43:04.486ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49602025-04-30T06:43:04.486ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49612025-04-30T06:43:04.486ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49622025-04-30T06:43:04.490ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49632025-04-30T06:43:04.498ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49642025-04-30T06:43:04.498ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49652025-04-30T06:43:04.498ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49662025-04-30T06:43:04.499ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49672025-04-30T06:43:04.568ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
49682025-04-30T06:43:04.569ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
49692025-04-30T06:43:04.570ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:34095 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
49702025-04-30T06:43:04.570ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:34095 req_id = 3dfc5c86-93d3-40ed-899e-599f92441e04 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49712025-04-30T06:43:04.571ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 468 local_addr = [::1]:51119 method = GET remote_addr = [::1]:34095 req_id = 3dfc5c86-93d3-40ed-899e-599f92441e04 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49722025-04-30T06:43:04.571ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49732025-04-30T06:43:04.571ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49742025-04-30T06:43:04.572ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49752025-04-30T06:43:04.572ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dfc5c86-93d3-40ed-899e-599f92441e04", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
49762025-04-30T06:43:04.574ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49772025-04-30T06:43:04.574ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49782025-04-30T06:43:04.574ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
49792025-04-30T06:43:04.577ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49802025-04-30T06:43:04.577ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49812025-04-30T06:43:04.577ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49822025-04-30T06:43:04.577ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49832025-04-30T06:43:04.578ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49842025-04-30T06:43:04.582ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49852025-04-30T06:43:04.589ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49862025-04-30T06:43:04.589ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49872025-04-30T06:43:04.589ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49882025-04-30T06:43:04.590ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49892025-04-30T06:43:04.632ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
49902025-04-30T06:43:04.633ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
49912025-04-30T06:43:04.633ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 470.520591ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49922025-04-30T06:43:04.633ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
49932025-04-30T06:43:04.633ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49942025-04-30T06:43:04.633ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49952025-04-30T06:43:04.633ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49962025-04-30T06:43:04.634ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49972025-04-30T06:43:04.635ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49982025-04-30T06:43:04.639ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49992025-04-30T06:43:04.645ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
50002025-04-30T06:43:04.645ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50012025-04-30T06:43:04.645ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50022025-04-30T06:43:04.646ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50032025-04-30T06:43:04.652ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
50042025-04-30T06:43:04.653ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
50052025-04-30T06:43:04.653ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
50062025-04-30T06:43:04.655ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ff bf 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63859 req_id = 499b33e6-2240-48ab-a7a9-dbb760420172
50072025-04-30T06:43:04.655ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65471,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63859 req_id = 499b33e6-2240-48ab-a7a9-dbb760420172
50082025-04-30T06:43:04.655ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
50092025-04-30T06:43:04.655ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
50102025-04-30T06:43:04.656ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63859 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 499b33e6-2240-48ab-a7a9-dbb760420172
50112025-04-30T06:43:04.656ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4249, tv_nsec: 455413330 } })
50122025-04-30T06:43:04.656ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
50132025-04-30T06:43:04.656ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2c d8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ,...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61295 req_id = 51dee2cd-6355-4e64-a06c-4deab4281ed1
50142025-04-30T06:43:04.657ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11480,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61295 req_id = 51dee2cd-6355-4e64-a06c-4deab4281ed1
50152025-04-30T06:43:04.657ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
50162025-04-30T06:43:04.657ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61295 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 51dee2cd-6355-4e64-a06c-4deab4281ed1
50172025-04-30T06:43:04.738ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
50182025-04-30T06:43:04.738ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50192025-04-30T06:43:04.738ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50202025-04-30T06:43:04.738ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50212025-04-30T06:43:04.738ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
50222025-04-30T06:43:04.740ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50232025-04-30T06:43:04.740ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50242025-04-30T06:43:04.740ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50252025-04-30T06:43:04.740ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50262025-04-30T06:43:04.740ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50272025-04-30T06:43:04.741ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
50282025-04-30T06:43:04.742ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50292025-04-30T06:43:04.743ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d2 eb 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63304 req_id = 2e75af2d-e5a6-4911-8bff-f2a4d8075907
50302025-04-30T06:43:04.743ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53995,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63304 req_id = 2e75af2d-e5a6-4911-8bff-f2a4d8075907
50312025-04-30T06:43:04.743ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
50322025-04-30T06:43:04.744ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53995, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63304 req_id = 2e75af2d-e5a6-4911-8bff-f2a4d8075907
50332025-04-30T06:43:04.745ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50342025-04-30T06:43:04.754ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50352025-04-30T06:43:04.754ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50362025-04-30T06:43:04.754ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
50372025-04-30T06:43:04.767ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50382025-04-30T06:43:04.767ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
50392025-04-30T06:43:04.767ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
50402025-04-30T06:43:04.768ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
50412025-04-30T06:43:04.768ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50422025-04-30T06:43:04.770ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50432025-04-30T06:43:04.770ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50442025-04-30T06:43:04.770ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50452025-04-30T06:43:04.771ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
50462025-04-30T06:43:04.771ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50472025-04-30T06:43:04.773ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "df8c604d-a332-4b85-a88c-7620e25439ba", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50482025-04-30T06:43:04.773ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
50492025-04-30T06:43:04.775ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
50502025-04-30T06:43:04.775ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50512025-04-30T06:43:04.775ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50522025-04-30T06:43:04.775ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
50532025-04-30T06:43:04.776ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50542025-04-30T06:43:04.776ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cd5192f-4424-4a9a-b104-3e343a8c24ff", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50552025-04-30T06:43:04.777ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
50562025-04-30T06:43:04.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69d5f06e-cb9d-480b-ad25-159302492938", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50572025-04-30T06:43:04.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
50582025-04-30T06:43:04.783ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "315d30ef-dc11-48eb-a1e4-96bc8639aee2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50592025-04-30T06:43:04.783ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
50602025-04-30T06:43:04.785ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dce83c5a-39f7-4c80-a99a-30015f84c19a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50612025-04-30T06:43:04.785ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
50622025-04-30T06:43:04.787ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bb85714-fe6c-4f53-832d-8d0bb87df0fa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50632025-04-30T06:43:04.787ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
50642025-04-30T06:43:04.789ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb5b065f-53ea-4c8e-9fea-45f0c2d0f68a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50652025-04-30T06:43:04.790ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
50662025-04-30T06:43:04.797ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6fa11f5-c317-4cef-a823-15cd5ad0b4ac", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50672025-04-30T06:43:04.798ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
50682025-04-30T06:43:04.799ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e57456d4-e5c1-4e03-a55e-05254a785c8a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50692025-04-30T06:43:04.799ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
50702025-04-30T06:43:04.801ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4ecd6fa-fb44-4b04-b41b-6519f89f06cf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50712025-04-30T06:43:04.801ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
50722025-04-30T06:43:04.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98b65a7c-479f-497e-980e-277e26abeb1b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50732025-04-30T06:43:04.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
50742025-04-30T06:43:04.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca4a016-19c7-44e9-9bc4-3aeba4bcdef3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50752025-04-30T06:43:04.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
50762025-04-30T06:43:04.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ab2df8-b090-4a7e-890a-3b081bd041f0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50772025-04-30T06:43:04.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
50782025-04-30T06:43:04.810ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
50792025-04-30T06:43:04.812ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "189ed495-403e-4e98-8669-d2303bc09f66", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50802025-04-30T06:43:04.812ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
50812025-04-30T06:43:04.815ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b39a27d-0e7e-4f96-85be-6f60f9815abc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50822025-04-30T06:43:04.815ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
50832025-04-30T06:43:04.817ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eb8e5b5-ccaa-4758-88d3-fddd052e3ad9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50842025-04-30T06:43:04.817ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
50852025-04-30T06:43:04.820ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b24b761-a02b-49f1-b451-10eb7de0f688", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50862025-04-30T06:43:04.821ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
50872025-04-30T06:43:04.824ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59cb4cc1-5288-42b2-8ec4-4e0d91e68d15", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50882025-04-30T06:43:04.824ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
50892025-04-30T06:43:04.826ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e85a2785-dcdc-4a51-b345-2157762622db", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50902025-04-30T06:43:04.826ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
50912025-04-30T06:43:04.827ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca50b3de-e7b0-4a63-ac29-b012b3a6c538", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50922025-04-30T06:43:04.827ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
50932025-04-30T06:43:04.829ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95336af9-00dc-4c72-a512-2f1da5470111", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50942025-04-30T06:43:04.829ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
50952025-04-30T06:43:04.830ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df92ec89-bcdd-47a2-9ecd-b09e5dd87d3a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50962025-04-30T06:43:04.830ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
50972025-04-30T06:43:04.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c74c344-88fd-429e-af1c-8dec0b7c410f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
50982025-04-30T06:43:04.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
50992025-04-30T06:43:04.836ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "539c51db-766d-4fa7-8f04-2fc5d033e9c9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51002025-04-30T06:43:04.836ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
51012025-04-30T06:43:04.839ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4015483-deb4-44cc-a3eb-40c70bb585d3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51022025-04-30T06:43:04.839ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
51032025-04-30T06:43:04.842ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
51042025-04-30T06:43:04.843ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55bdb301-c5b8-4e96-ac10-a0b341a9d57f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51052025-04-30T06:43:04.844ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
51062025-04-30T06:43:04.846ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca03d713-19d9-4d2c-8bb5-e0748113cbe2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51072025-04-30T06:43:04.847ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
51082025-04-30T06:43:04.851ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4297c896-0a35-4929-aeb5-c754e91415fd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51092025-04-30T06:43:04.851ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
51102025-04-30T06:43:04.853ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fc1b6ba-d144-48ae-b517-280ef4bcc417", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51112025-04-30T06:43:04.853ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
51122025-04-30T06:43:04.856ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "770d18a6-6667-4b39-af92-b2e124a3dcf5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51132025-04-30T06:43:04.856ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
51142025-04-30T06:43:04.857ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "099038a7-e747-4ae0-a097-4ae3b1de7ba1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51152025-04-30T06:43:04.857ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
51162025-04-30T06:43:04.859ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15bd97a5-e018-4636-b8f3-eba54161e790", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51172025-04-30T06:43:04.859ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
51182025-04-30T06:43:04.861ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b3eb88d-4e31-42bc-bd79-e8e930965cc9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51192025-04-30T06:43:04.861ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
51202025-04-30T06:43:04.861ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51212025-04-30T06:43:04.861ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51222025-04-30T06:43:04.861ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51232025-04-30T06:43:04.861ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
51242025-04-30T06:43:04.862ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51252025-04-30T06:43:04.864ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a7ae7c4-c7b4-4168-a760-9cb29c044bf0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51262025-04-30T06:43:04.864ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
51272025-04-30T06:43:04.865ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51282025-04-30T06:43:04.873ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51292025-04-30T06:43:04.873ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51302025-04-30T06:43:04.873ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
51312025-04-30T06:43:04.874ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51322025-04-30T06:43:04.877ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8aecd7d-966e-4eab-bfb2-223185f585c6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51332025-04-30T06:43:04.877ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
51342025-04-30T06:43:04.878ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51352025-04-30T06:43:04.883ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
51362025-04-30T06:43:04.886ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51372025-04-30T06:43:04.891ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8879072e-f405-4e76-a096-f7fb6570e13f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51382025-04-30T06:43:04.891ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
51392025-04-30T06:43:04.904ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e5317ea-ea10-4f7b-9224-92c48ce0739e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51402025-04-30T06:43:04.904ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
51412025-04-30T06:43:04.911ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13daea1e-5fad-42af-8bf9-115cf9ca0433", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51422025-04-30T06:43:04.912ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
51432025-04-30T06:43:04.916ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61876692-3a05-4269-bd4a-d31cd78c0f1c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51442025-04-30T06:43:04.916ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
51452025-04-30T06:43:04.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea61e84c-53c0-459e-907c-aafec8009e4c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51462025-04-30T06:43:04.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
51472025-04-30T06:43:04.924ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e20df981-ef23-4466-8284-3b3fd3222bad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51482025-04-30T06:43:04.924ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
51492025-04-30T06:43:04.930ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37b602dd-aae9-4046-a9a6-9197a2b4683f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51502025-04-30T06:43:04.930ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
51512025-04-30T06:43:04.931ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
51522025-04-30T06:43:04.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7e2367e-fc3d-4e58-adba-5fde929b5f60", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51532025-04-30T06:43:04.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
51542025-04-30T06:43:04.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76d300b9-f6ea-4d23-949a-30a5d23224a4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51552025-04-30T06:43:04.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
51562025-04-30T06:43:04.937ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4875d673-87d2-40e9-9a0f-9ede069e4c5e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51572025-04-30T06:43:04.938ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
51582025-04-30T06:43:04.938ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51592025-04-30T06:43:04.938ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51602025-04-30T06:43:04.938ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51612025-04-30T06:43:04.938ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51622025-04-30T06:43:04.941ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51632025-04-30T06:43:04.946ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51642025-04-30T06:43:04.946ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
51652025-04-30T06:43:04.946ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51662025-04-30T06:43:04.946ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51672025-04-30T06:43:04.946ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51682025-04-30T06:43:04.947ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51692025-04-30T06:43:04.950ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51702025-04-30T06:43:04.961ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51712025-04-30T06:43:04.961ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51722025-04-30T06:43:04.961ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51732025-04-30T06:43:04.962ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51742025-04-30T06:43:05.065ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
51752025-04-30T06:43:05.066ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
51762025-04-30T06:43:05.073ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51772025-04-30T06:43:05.088ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51782025-04-30T06:43:05.088ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51792025-04-30T06:43:05.088ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
51802025-04-30T06:43:05.089ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51812025-04-30T06:43:05.090ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:60037 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
51822025-04-30T06:43:05.093ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:60037 req_id = 65ff6043-4a96-4e0b-8a14-75e43a75d5ba sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51832025-04-30T06:43:05.094ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 658 local_addr = [::1]:51119 method = GET remote_addr = [::1]:60037 req_id = 65ff6043-4a96-4e0b-8a14-75e43a75d5ba response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51842025-04-30T06:43:05.094ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "65ff6043-4a96-4e0b-8a14-75e43a75d5ba", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
51852025-04-30T06:43:05.097ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51862025-04-30T06:43:05.097ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51872025-04-30T06:43:05.097ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
51882025-04-30T06:43:05.100ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51892025-04-30T06:43:05.100ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51902025-04-30T06:43:05.100ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51912025-04-30T06:43:05.100ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51922025-04-30T06:43:05.101ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51932025-04-30T06:43:05.104ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51942025-04-30T06:43:05.110ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51952025-04-30T06:43:05.110ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51962025-04-30T06:43:05.110ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
51972025-04-30T06:43:05.113ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51982025-04-30T06:43:05.139ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51992025-04-30T06:43:05.146ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
52002025-04-30T06:43:05.148ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52012025-04-30T06:43:05.148ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52022025-04-30T06:43:05.149ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52032025-04-30T06:43:05.155ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52042025-04-30T06:43:05.157ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52052025-04-30T06:43:05.157ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52062025-04-30T06:43:05.260ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
52072025-04-30T06:43:05.260ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
52082025-04-30T06:43:05.261ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 627.387057ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
52092025-04-30T06:43:05.261ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
52102025-04-30T06:43:05.261ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
52112025-04-30T06:43:05.261ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52122025-04-30T06:43:05.261ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52132025-04-30T06:43:05.263ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52142025-04-30T06:43:05.264ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52152025-04-30T06:43:05.268ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52162025-04-30T06:43:05.272ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
52172025-04-30T06:43:05.272ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52182025-04-30T06:43:05.272ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52192025-04-30T06:43:05.273ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52202025-04-30T06:43:05.285ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
52212025-04-30T06:43:05.288ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
52222025-04-30T06:43:05.288ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
52232025-04-30T06:43:05.289ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bd 39 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .9..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49545 req_id = 1990725e-89f7-4fdb-940a-eea3f7c42fd2
52242025-04-30T06:43:05.289ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48441,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49545 req_id = 1990725e-89f7-4fdb-940a-eea3f7c42fd2
52252025-04-30T06:43:05.290ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
52262025-04-30T06:43:05.290ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
52272025-04-30T06:43:05.290ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49545 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1990725e-89f7-4fdb-940a-eea3f7c42fd2
52282025-04-30T06:43:05.290ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4250, tv_nsec: 89673972 } })
52292025-04-30T06:43:05.290ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
52302025-04-30T06:43:05.291ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 00 e5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53132 req_id = 02ef097f-eea5-47dc-b393-fc8682618e45
52312025-04-30T06:43:05.291ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 229,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53132 req_id = 02ef097f-eea5-47dc-b393-fc8682618e45
52322025-04-30T06:43:05.291ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
52332025-04-30T06:43:05.291ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53132 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 02ef097f-eea5-47dc-b393-fc8682618e45
52342025-04-30T06:43:05.343ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
52352025-04-30T06:43:05.344ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52362025-04-30T06:43:05.344ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52372025-04-30T06:43:05.344ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52382025-04-30T06:43:05.344ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52392025-04-30T06:43:05.344ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52402025-04-30T06:43:05.345ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ca 11 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61808 req_id = 405a1f88-71dd-49fb-872c-2ea47fdf0354
52412025-04-30T06:43:05.345ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51729,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61808 req_id = 405a1f88-71dd-49fb-872c-2ea47fdf0354
52422025-04-30T06:43:05.345ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
52432025-04-30T06:43:05.345ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51729, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61808 req_id = 405a1f88-71dd-49fb-872c-2ea47fdf0354
52442025-04-30T06:43:05.345ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
52452025-04-30T06:43:05.345ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
52462025-04-30T06:43:05.345ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
52472025-04-30T06:43:05.347ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d8dbdf3-328d-4d00-9080-87a60cec07e1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52482025-04-30T06:43:05.348ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
52492025-04-30T06:43:05.348ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52502025-04-30T06:43:05.353ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52512025-04-30T06:43:05.353ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52522025-04-30T06:43:05.353ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52532025-04-30T06:43:05.354ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52542025-04-30T06:43:05.356ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "323a676e-713f-47ee-99f6-29a615fbffd3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52552025-04-30T06:43:05.356ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
52562025-04-30T06:43:05.356ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52572025-04-30T06:43:05.356ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52582025-04-30T06:43:05.357ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52592025-04-30T06:43:05.361ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4be6ac1e-5dde-4fdb-8648-6bdad7dd02f4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52602025-04-30T06:43:05.361ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
52612025-04-30T06:43:05.364ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c5d0e1b-c46d-46ca-84ba-174b230db1db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52622025-04-30T06:43:05.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
52632025-04-30T06:43:05.386ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cf47e53-410b-4cc1-9e40-e3ad66f66439", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52642025-04-30T06:43:05.386ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
52652025-04-30T06:43:05.398ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "035ea5c7-dfee-460b-8f3e-5a9a67a7dd08", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52662025-04-30T06:43:05.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
52672025-04-30T06:43:05.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34b073c0-5db5-434a-a020-dbcc067ed79e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52682025-04-30T06:43:05.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
52692025-04-30T06:43:05.409ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c55fbf83-be17-459d-86d8-ef39edd3f476", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52702025-04-30T06:43:05.409ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
52712025-04-30T06:43:05.412ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e6f943c-4435-4d2c-a88a-8965e77968b8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52722025-04-30T06:43:05.413ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
52732025-04-30T06:43:05.423ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
52742025-04-30T06:43:05.424ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4f21c7b-45a9-49d8-b4fd-19348a736eba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52752025-04-30T06:43:05.431ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
52762025-04-30T06:43:05.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1def5fa7-ce7c-4114-b4d3-ce7ea47c283d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52772025-04-30T06:43:05.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
52782025-04-30T06:43:05.451ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c32c61b0-e58a-4b6e-8a72-a5f8f935c094", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52792025-04-30T06:43:05.452ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
52802025-04-30T06:43:05.456ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42ed9281-a08c-4a3e-8d06-a0252117ddb8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52812025-04-30T06:43:05.457ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
52822025-04-30T06:43:05.463ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2529487d-a2ed-4b7e-8dbc-1ba3c1d50fb9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52832025-04-30T06:43:05.465ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
52842025-04-30T06:43:05.469ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ab625aa-8e27-4c60-9091-630b5726937b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52852025-04-30T06:43:05.469ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
52862025-04-30T06:43:05.474ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52872025-04-30T06:43:05.474ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52882025-04-30T06:43:05.474ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52892025-04-30T06:43:05.474ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52902025-04-30T06:43:05.475ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52912025-04-30T06:43:05.476ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71c69b7c-32b2-4dc5-ab53-d8b5a69b2f9c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
52922025-04-30T06:43:05.477ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
52932025-04-30T06:43:05.479ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52942025-04-30T06:43:05.493ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52952025-04-30T06:43:05.493ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52962025-04-30T06:43:05.493ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52972025-04-30T06:43:05.493ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52982025-04-30T06:43:05.496ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52992025-04-30T06:43:05.496ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
53002025-04-30T06:43:05.497ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53012025-04-30T06:43:05.497ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "096c490b-9b0f-483d-b2f0-23aa240cde07", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53022025-04-30T06:43:05.497ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
53032025-04-30T06:43:05.502ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bc4f63a-59f8-4092-9cee-1b7c4fa672a2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53042025-04-30T06:43:05.502ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
53052025-04-30T06:43:05.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2eea96a-0a12-4b16-93aa-8a51f2f4abeb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53062025-04-30T06:43:05.506ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
53072025-04-30T06:43:05.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fee43af2-aa6b-4d6e-ba3b-a79117304edd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53082025-04-30T06:43:05.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
53092025-04-30T06:43:05.513ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fb60660-8944-4d6c-9bac-0d3e4ccae707", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53102025-04-30T06:43:05.513ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
53112025-04-30T06:43:05.516ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "626426ab-0775-4e6f-855d-c2fa1b8c75f4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53122025-04-30T06:43:05.516ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
53132025-04-30T06:43:05.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "68237d1d-d75f-40a2-9d5f-3996a52781ed", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53142025-04-30T06:43:05.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
53152025-04-30T06:43:05.519ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7574b4de-30a9-4ed6-98f5-b245d0132394", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53162025-04-30T06:43:05.519ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
53172025-04-30T06:43:05.523ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6d5d0cb-d87e-4a56-b60e-c39d4e3d8295", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53182025-04-30T06:43:05.523ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
53192025-04-30T06:43:05.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc40fbbb-6e61-4ad3-9f91-9c23f1086903", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53202025-04-30T06:43:05.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
53212025-04-30T06:43:05.561ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
53222025-04-30T06:43:05.563ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9dad8fc-6a0f-498e-8d4f-25458e5eaaa0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53232025-04-30T06:43:05.563ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
53242025-04-30T06:43:05.566ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48caeb01-fac7-4ce9-acc6-6746ab769cd9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53252025-04-30T06:43:05.567ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
53262025-04-30T06:43:05.582ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
53272025-04-30T06:43:05.582ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11eb3f44-9948-4152-8be8-1ad8034890c6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53282025-04-30T06:43:05.582ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
53292025-04-30T06:43:05.582ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53302025-04-30T06:43:05.583ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 5638f445-209f-412e-8f9d-59e0146287aa uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53312025-04-30T06:43:05.583ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 5638f445-209f-412e-8f9d-59e0146287aa resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53322025-04-30T06:43:05.583ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 5638f445-209f-412e-8f9d-59e0146287aa roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53332025-04-30T06:43:05.584ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 5638f445-209f-412e-8f9d-59e0146287aa resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53342025-04-30T06:43:05.588ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32308f76-7658-41a0-8113-f9b710d75516", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53352025-04-30T06:43:05.588ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
53362025-04-30T06:43:05.588ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 5784 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 5638f445-209f-412e-8f9d-59e0146287aa response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53372025-04-30T06:43:05.589ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "5638f445-209f-412e-8f9d-59e0146287aa", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53382025-04-30T06:43:05.589ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53392025-04-30T06:43:05.590ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 897ee3a8-a69b-4668-a096-95e8dba488b6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53402025-04-30T06:43:05.590ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 897ee3a8-a69b-4668-a096-95e8dba488b6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53412025-04-30T06:43:05.590ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 897ee3a8-a69b-4668-a096-95e8dba488b6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53422025-04-30T06:43:05.596ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 897ee3a8-a69b-4668-a096-95e8dba488b6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53432025-04-30T06:43:05.596ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9551e04-889a-4702-9a3d-b48a1cede6e2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53442025-04-30T06:43:05.596ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
53452025-04-30T06:43:05.596ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 4493 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 897ee3a8-a69b-4668-a096-95e8dba488b6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53462025-04-30T06:43:05.596ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "897ee3a8-a69b-4668-a096-95e8dba488b6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53472025-04-30T06:43:05.596ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
53482025-04-30T06:43:05.596ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53492025-04-30T06:43:05.596ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
53502025-04-30T06:43:05.596ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f671ed8d-b468-45a7-9d3e-99c373c48edd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53512025-04-30T06:43:05.596ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
53522025-04-30T06:43:05.597ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "843ff511-c9bd-4a85-8319-37206705bc1b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53532025-04-30T06:43:05.597ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
53542025-04-30T06:43:05.599ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a13c38f7-fd49-4c82-ac0b-d5f3c65c6535", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53552025-04-30T06:43:05.599ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
53562025-04-30T06:43:05.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e46dbff0-978c-4f2f-8408-8eaed930b78d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53572025-04-30T06:43:05.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
53582025-04-30T06:43:05.616ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53592025-04-30T06:43:05.616ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53602025-04-30T06:43:05.616ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53612025-04-30T06:43:05.616ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
53622025-04-30T06:43:05.619ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53632025-04-30T06:43:05.619ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ecb114d-7562-4079-8e1b-a056127cad63", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53642025-04-30T06:43:05.619ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
53652025-04-30T06:43:05.622ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53662025-04-30T06:43:05.631ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53672025-04-30T06:43:05.631ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53682025-04-30T06:43:05.632ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
53692025-04-30T06:43:05.632ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53702025-04-30T06:43:05.635ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53712025-04-30T06:43:05.635ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
53722025-04-30T06:43:05.636ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53732025-04-30T06:43:05.646ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36344775-86e0-4e84-8359-60124ee3687a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53742025-04-30T06:43:05.647ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
53752025-04-30T06:43:05.653ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f023a529-4851-4b4b-9dff-b1ab6a3f86fb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53762025-04-30T06:43:05.653ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
53772025-04-30T06:43:05.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "382d2919-4624-4018-a9e9-4655e1588da6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53782025-04-30T06:43:05.658ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
53792025-04-30T06:43:05.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7a15cc0-61bb-49ee-9de1-c98b5b8d838f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53802025-04-30T06:43:05.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
53812025-04-30T06:43:05.669ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0132393b-3eff-4840-b227-08efa953e495", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53822025-04-30T06:43:05.670ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
53832025-04-30T06:43:05.671ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46f13d2f-490a-46a0-9152-ff08904973bd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53842025-04-30T06:43:05.671ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
53852025-04-30T06:43:05.675ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33d944fa-53cb-49b5-920c-aa10b5f28a99", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53862025-04-30T06:43:05.675ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
53872025-04-30T06:43:05.678ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29cf1113-278c-4a8b-8399-cdb0bab2b12c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53882025-04-30T06:43:05.678ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
53892025-04-30T06:43:05.687ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53902025-04-30T06:43:05.687ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53912025-04-30T06:43:05.687ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c645fb4e-e274-456e-8b70-900c457c3802", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
53922025-04-30T06:43:05.687ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53932025-04-30T06:43:05.687ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53942025-04-30T06:43:05.687ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53952025-04-30T06:43:05.687ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53962025-04-30T06:43:05.689ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53972025-04-30T06:43:05.690ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
53982025-04-30T06:43:05.692ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
53992025-04-30T06:43:05.701ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
54002025-04-30T06:43:05.708ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54012025-04-30T06:43:05.714ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
54022025-04-30T06:43:05.714ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
54032025-04-30T06:43:05.714ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54042025-04-30T06:43:05.714ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54052025-04-30T06:43:05.714ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54062025-04-30T06:43:05.715ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54072025-04-30T06:43:05.717ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
54082025-04-30T06:43:05.717ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
54092025-04-30T06:43:05.723ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54102025-04-30T06:43:05.731ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
54112025-04-30T06:43:05.731ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54122025-04-30T06:43:05.731ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54132025-04-30T06:43:05.732ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54142025-04-30T06:43:05.734ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
54152025-04-30T06:43:05.734ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54162025-04-30T06:43:05.734ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54172025-04-30T06:43:05.734ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54182025-04-30T06:43:05.739ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54192025-04-30T06:43:05.741ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54202025-04-30T06:43:05.741ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54212025-04-30T06:43:05.741ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54222025-04-30T06:43:05.829ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
54232025-04-30T06:43:05.830ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
54242025-04-30T06:43:05.831ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54252025-04-30T06:43:05.831ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54262025-04-30T06:43:05.833ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54272025-04-30T06:43:05.833ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
54282025-04-30T06:43:05.884ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54292025-04-30T06:43:05.912ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:58922 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
54302025-04-30T06:43:05.913ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
54312025-04-30T06:43:05.913ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:58922 req_id = 63aef481-46d5-49e3-a053-f054df55b2c6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
54322025-04-30T06:43:05.914ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 491 local_addr = [::1]:51119 method = GET remote_addr = [::1]:58922 req_id = 63aef481-46d5-49e3-a053-f054df55b2c6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
54332025-04-30T06:43:05.914ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "63aef481-46d5-49e3-a053-f054df55b2c6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
54342025-04-30T06:43:05.916ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
54352025-04-30T06:43:05.916ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
54362025-04-30T06:43:05.916ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
54372025-04-30T06:43:05.918ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
54382025-04-30T06:43:05.918ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54392025-04-30T06:43:05.918ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54402025-04-30T06:43:05.918ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54412025-04-30T06:43:05.919ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54422025-04-30T06:43:05.919ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54432025-04-30T06:43:05.942ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54442025-04-30T06:43:05.942ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54452025-04-30T06:43:05.942ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
54462025-04-30T06:43:05.942ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54472025-04-30T06:43:05.959ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54482025-04-30T06:43:05.959ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
54492025-04-30T06:43:05.960ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54502025-04-30T06:43:05.960ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54512025-04-30T06:43:05.964ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
54522025-04-30T06:43:05.965ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54532025-04-30T06:43:05.965ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54542025-04-30T06:43:05.965ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54552025-04-30T06:43:06.052ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
54562025-04-30T06:43:06.052ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
54572025-04-30T06:43:06.052ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 791.116758ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
54582025-04-30T06:43:06.052ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
54592025-04-30T06:43:06.052ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
54602025-04-30T06:43:06.052ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54612025-04-30T06:43:06.052ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54622025-04-30T06:43:06.052ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54632025-04-30T06:43:06.053ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54642025-04-30T06:43:06.054ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
54652025-04-30T06:43:06.057ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54662025-04-30T06:43:06.061ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
54672025-04-30T06:43:06.061ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54682025-04-30T06:43:06.061ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54692025-04-30T06:43:06.061ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54702025-04-30T06:43:06.065ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
54712025-04-30T06:43:06.065ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
54722025-04-30T06:43:06.066ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 12 01 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60541 req_id = 43cbb92b-787c-4b56-9ab7-552b264fbf4e
54732025-04-30T06:43:06.066ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4609,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60541 req_id = 43cbb92b-787c-4b56-9ab7-552b264fbf4e
54742025-04-30T06:43:06.066ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
54752025-04-30T06:43:06.067ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54762025-04-30T06:43:06.067ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60541 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 43cbb92b-787c-4b56-9ab7-552b264fbf4e
54772025-04-30T06:43:06.067ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4250, tv_nsec: 866467041 } })
54782025-04-30T06:43:06.067ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
54792025-04-30T06:43:06.067ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b7 21 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .!..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62717 req_id = 924c9c95-4596-4783-83ea-92bcadd4848d
54802025-04-30T06:43:06.067ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46881,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62717 req_id = 924c9c95-4596-4783-83ea-92bcadd4848d
54812025-04-30T06:43:06.068ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54822025-04-30T06:43:06.068ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62717 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 924c9c95-4596-4783-83ea-92bcadd4848d
54832025-04-30T06:43:06.209ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
54842025-04-30T06:43:06.209ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54852025-04-30T06:43:06.209ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54862025-04-30T06:43:06.209ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54872025-04-30T06:43:06.214ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
54882025-04-30T06:43:06.215ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54892025-04-30T06:43:06.218ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c2 da 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52746 req_id = fafb4a8b-fc36-4e87-b83a-d15530e708c1
54902025-04-30T06:43:06.218ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49882,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52746 req_id = fafb4a8b-fc36-4e87-b83a-d15530e708c1
54912025-04-30T06:43:06.218ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
54922025-04-30T06:43:06.218ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49882, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52746 req_id = fafb4a8b-fc36-4e87-b83a-d15530e708c1
54932025-04-30T06:43:06.219ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
54942025-04-30T06:43:06.219ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
54952025-04-30T06:43:06.219ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
54962025-04-30T06:43:06.219ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54972025-04-30T06:43:06.224ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54982025-04-30T06:43:06.224ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54992025-04-30T06:43:06.224ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55002025-04-30T06:43:06.225ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55012025-04-30T06:43:06.235ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55022025-04-30T06:43:06.235ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55032025-04-30T06:43:06.236ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55042025-04-30T06:43:06.240ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e697327e-e441-4e56-93f2-ccf862f30009", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55052025-04-30T06:43:06.240ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
55062025-04-30T06:43:06.254ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a91c2b4-8f82-4353-a137-f77b678768a2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55072025-04-30T06:43:06.256ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
55082025-04-30T06:43:06.263ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a7033d3-d782-47df-9943-b3f77620d43d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55092025-04-30T06:43:06.263ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
55102025-04-30T06:43:06.275ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df88fe92-36dd-48e8-bbfe-a1dd2e244545", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55112025-04-30T06:43:06.275ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
55122025-04-30T06:43:06.282ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2785eb10-d8a0-4325-bf65-2b9e0b86336c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55132025-04-30T06:43:06.283ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
55142025-04-30T06:43:06.316ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
55152025-04-30T06:43:06.318ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f12d623-3f7c-4c9a-9548-264c3ea8d2ca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55162025-04-30T06:43:06.319ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
55172025-04-30T06:43:06.330ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8ea4b17-1279-41ff-abf9-d80c24a70eb2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55182025-04-30T06:43:06.331ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
55192025-04-30T06:43:06.341ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54258a38-2ca6-4c16-83f6-12423ea8fd2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55202025-04-30T06:43:06.343ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
55212025-04-30T06:43:06.346ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e10cd42-3e17-4393-8be0-047c621021dc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55222025-04-30T06:43:06.346ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
55232025-04-30T06:43:06.348ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fe5f049-f2af-4a47-a0c9-1a0babe36851", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55242025-04-30T06:43:06.348ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
55252025-04-30T06:43:06.351ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23245fbd-9519-41e2-86d2-877a8c654720", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55262025-04-30T06:43:06.352ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
55272025-04-30T06:43:06.355ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdfe6c42-da53-4a0a-a34d-5f887211afdc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55282025-04-30T06:43:06.355ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
55292025-04-30T06:43:06.388ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55302025-04-30T06:43:06.388ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55312025-04-30T06:43:06.388ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55322025-04-30T06:43:06.391ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55332025-04-30T06:43:06.391ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55342025-04-30T06:43:06.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3bf8bff-ed3b-44ba-a0e2-e41eaed770de", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55352025-04-30T06:43:06.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
55362025-04-30T06:43:06.395ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55372025-04-30T06:43:06.424ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55382025-04-30T06:43:06.424ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55392025-04-30T06:43:06.424ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55402025-04-30T06:43:06.425ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55412025-04-30T06:43:06.428ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55422025-04-30T06:43:06.428ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55432025-04-30T06:43:06.429ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55442025-04-30T06:43:06.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "066d520c-da65-4b4c-8618-e38e16e14be5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55452025-04-30T06:43:06.434ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
55462025-04-30T06:43:06.443ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63145aa3-7581-4d9b-b179-ddbdadc3cda1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55472025-04-30T06:43:06.443ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
55482025-04-30T06:43:06.469ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db586988-fc4f-49d0-98f6-ad42a415e917", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55492025-04-30T06:43:06.469ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
55502025-04-30T06:43:06.475ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "794dc2f0-5c90-4e7d-91eb-fee0878f99fd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55512025-04-30T06:43:06.475ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
55522025-04-30T06:43:06.476ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
55532025-04-30T06:43:06.478ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "135a366b-d9ee-4648-b224-4f2cad1f1d63", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55542025-04-30T06:43:06.478ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
55552025-04-30T06:43:06.480ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7e50006-c042-4d7a-91e6-945bfb20c824", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55562025-04-30T06:43:06.480ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
55572025-04-30T06:43:06.481ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b231cba4-9d09-44d7-9f79-fb7d6657dd0a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55582025-04-30T06:43:06.482ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
55592025-04-30T06:43:06.483ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c461b91-80bc-4138-99c3-7ae411a4a596", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55602025-04-30T06:43:06.483ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
55612025-04-30T06:43:06.484ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bf0265b-eeb5-4959-be24-54b52d62abb3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55622025-04-30T06:43:06.484ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
55632025-04-30T06:43:06.485ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e11f1b31-55e5-44e1-94f1-48cd735ef4d3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55642025-04-30T06:43:06.485ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
55652025-04-30T06:43:06.488ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5105775-fd01-4cc9-9996-d21e8aa2a52c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55662025-04-30T06:43:06.488ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
55672025-04-30T06:43:06.491ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97d16b41-7038-4574-be9b-abe2915eb5ac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55682025-04-30T06:43:06.491ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
55692025-04-30T06:43:06.494ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8ae5814-7692-4c10-b225-51cd85af05f4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55702025-04-30T06:43:06.494ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
55712025-04-30T06:43:06.496ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "794dfd95-cec2-4c9e-a8c1-cbdb18d67093", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55722025-04-30T06:43:06.496ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
55732025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13498790-3b74-4400-a6a4-7dc7af863e67", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55742025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
55752025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "561906d3-5b93-4167-a006-569bb0b7624c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55762025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
55772025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbccc98e-0944-4d81-a48c-11cc6a4dae2b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55782025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
55792025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f460bd4-a813-4b1c-9587-d9b106402b8a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55802025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
55812025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95c381a0-6ee9-4668-832f-db7f5bbe9bcb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55822025-04-30T06:43:06.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
55832025-04-30T06:43:06.504ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e6a077b-71f3-470c-9761-cf2fb228a126", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55842025-04-30T06:43:06.504ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
55852025-04-30T06:43:06.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c4fa503-d739-48d7-9298-ecb3c15ebc00", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55862025-04-30T06:43:06.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
55872025-04-30T06:43:06.508ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa110494-bbf5-43b6-bf8b-0672fa7faf52", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55882025-04-30T06:43:06.509ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
55892025-04-30T06:43:06.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b8dda03-23cc-4aed-bfd2-3478135be62a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55902025-04-30T06:43:06.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
55912025-04-30T06:43:06.520ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0137119a-996b-4a14-9a64-949dfd29bc98", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55922025-04-30T06:43:06.520ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
55932025-04-30T06:43:06.524ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05ff15b7-9f30-4e99-a529-3f191c6324ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
55942025-04-30T06:43:06.524ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
55952025-04-30T06:43:06.529ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55962025-04-30T06:43:06.542ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55972025-04-30T06:43:06.542ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55982025-04-30T06:43:06.542ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55992025-04-30T06:43:06.543ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56002025-04-30T06:43:06.550ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10e7b5aa-efeb-4928-a9f1-4490b24f02ea", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
56012025-04-30T06:43:06.550ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
56022025-04-30T06:43:06.551ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56032025-04-30T06:43:06.570ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56042025-04-30T06:43:06.570ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56052025-04-30T06:43:06.570ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
56062025-04-30T06:43:06.571ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56072025-04-30T06:43:06.576ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56082025-04-30T06:43:06.576ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
56092025-04-30T06:43:06.576ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56102025-04-30T06:43:06.577ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68595222-fd6e-4d7f-a16e-c8cf622abaa8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
56112025-04-30T06:43:06.577ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
56122025-04-30T06:43:06.582ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84e2ed8f-5c6d-4bbc-970b-253335e676fa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
56132025-04-30T06:43:06.583ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
56142025-04-30T06:43:06.589ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0764493-020d-47c6-afd3-e58671ef0826", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
56152025-04-30T06:43:06.589ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
56162025-04-30T06:43:06.593ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50f30d05-d2e4-4e91-8c09-e314baf5a0d4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
56172025-04-30T06:43:06.593ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
56182025-04-30T06:43:06.596ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70063ca5-32ad-4a86-97f1-16fee5b03ffd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
56192025-04-30T06:43:06.596ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
56202025-04-30T06:43:06.598ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcc86011-e3e1-4add-adc9-1a121f74fafa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
56212025-04-30T06:43:06.600ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
56222025-04-30T06:43:06.600ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56232025-04-30T06:43:06.600ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56242025-04-30T06:43:06.600ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56252025-04-30T06:43:06.600ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56262025-04-30T06:43:06.618ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56272025-04-30T06:43:06.624ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
56282025-04-30T06:43:06.624ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
56292025-04-30T06:43:06.624ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56302025-04-30T06:43:06.624ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56312025-04-30T06:43:06.624ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56322025-04-30T06:43:06.625ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56332025-04-30T06:43:06.656ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56342025-04-30T06:43:06.665ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
56352025-04-30T06:43:06.665ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56362025-04-30T06:43:06.665ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56372025-04-30T06:43:06.678ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56382025-04-30T06:43:06.686ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
56392025-04-30T06:43:06.686ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
56402025-04-30T06:43:06.773ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
56412025-04-30T06:43:06.774ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
56422025-04-30T06:43:06.774ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
56432025-04-30T06:43:06.775ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
56442025-04-30T06:43:06.778ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
56452025-04-30T06:43:06.778ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:62181 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
56462025-04-30T06:43:06.779ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56472025-04-30T06:43:06.779ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56482025-04-30T06:43:06.797ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
56492025-04-30T06:43:06.797ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
56502025-04-30T06:43:06.799ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:62181 req_id = 9fb3f77c-50d9-4102-a08d-a10bfe297f18 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
56512025-04-30T06:43:06.801ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
56522025-04-30T06:43:06.803ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 3911 local_addr = [::1]:51119 method = GET remote_addr = [::1]:62181 req_id = 9fb3f77c-50d9-4102-a08d-a10bfe297f18 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
56532025-04-30T06:43:06.806ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56542025-04-30T06:43:06.808ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56552025-04-30T06:43:06.808ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56562025-04-30T06:43:06.809ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56572025-04-30T06:43:06.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fb3f77c-50d9-4102-a08d-a10bfe297f18", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
56582025-04-30T06:43:06.813ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
56592025-04-30T06:43:06.827ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
56602025-04-30T06:43:06.827ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
56612025-04-30T06:43:06.833ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
56622025-04-30T06:43:06.834ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56632025-04-30T06:43:06.834ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56642025-04-30T06:43:06.834ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56652025-04-30T06:43:06.835ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56662025-04-30T06:43:06.836ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56672025-04-30T06:43:06.836ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56682025-04-30T06:43:06.836ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56692025-04-30T06:43:06.836ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
56702025-04-30T06:43:06.840ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56712025-04-30T06:43:06.840ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
56722025-04-30T06:43:06.840ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
56732025-04-30T06:43:06.859ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56742025-04-30T06:43:06.865ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56752025-04-30T06:43:06.865ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56762025-04-30T06:43:06.866ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
56772025-04-30T06:43:06.866ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56782025-04-30T06:43:06.866ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56792025-04-30T06:43:06.873ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
56802025-04-30T06:43:06.874ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56812025-04-30T06:43:06.874ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56822025-04-30T06:43:06.874ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56832025-04-30T06:43:06.884ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56842025-04-30T06:43:06.884ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
56852025-04-30T06:43:06.885ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56862025-04-30T06:43:06.965ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
56872025-04-30T06:43:06.965ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
56882025-04-30T06:43:06.965ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 912.909893ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
56892025-04-30T06:43:06.965ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
56902025-04-30T06:43:06.965ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
56912025-04-30T06:43:06.965ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56922025-04-30T06:43:06.965ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56932025-04-30T06:43:06.965ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56942025-04-30T06:43:06.966ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56952025-04-30T06:43:06.967ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
56962025-04-30T06:43:06.969ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56972025-04-30T06:43:06.993ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
56982025-04-30T06:43:06.993ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56992025-04-30T06:43:06.993ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57002025-04-30T06:43:06.994ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57012025-04-30T06:43:07.018ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57022025-04-30T06:43:07.018ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57032025-04-30T06:43:07.018ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57042025-04-30T06:43:07.018ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57052025-04-30T06:43:07.019ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57062025-04-30T06:43:07.026ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
57072025-04-30T06:43:07.026ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
57082025-04-30T06:43:07.038ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57092025-04-30T06:43:07.045ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57102025-04-30T06:43:07.045ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57112025-04-30T06:43:07.045ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57122025-04-30T06:43:07.047ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57132025-04-30T06:43:07.050ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 28 11 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 (............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50359 req_id = 83c59bbc-213a-4a89-8ef4-17d2a4383765
57142025-04-30T06:43:07.057ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10257,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50359 req_id = 83c59bbc-213a-4a89-8ef4-17d2a4383765
57152025-04-30T06:43:07.057ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
57162025-04-30T06:43:07.057ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57172025-04-30T06:43:07.058ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50359 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 83c59bbc-213a-4a89-8ef4-17d2a4383765
57182025-04-30T06:43:07.059ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4251, tv_nsec: 858681322 } })
57192025-04-30T06:43:07.059ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
57202025-04-30T06:43:07.065ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57212025-04-30T06:43:07.065ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57222025-04-30T06:43:07.066ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57232025-04-30T06:43:07.066ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c9 63 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .c..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62658 req_id = d4c9ff6f-200d-4e31-ae7e-68d8a7514834
57242025-04-30T06:43:07.066ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51555,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62658 req_id = d4c9ff6f-200d-4e31-ae7e-68d8a7514834
57252025-04-30T06:43:07.066ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57262025-04-30T06:43:07.066ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62658 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d4c9ff6f-200d-4e31-ae7e-68d8a7514834
57272025-04-30T06:43:07.144ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
57282025-04-30T06:43:07.151ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9a d6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51858 req_id = 023db374-477c-45f6-a109-7fa44999779e
57292025-04-30T06:43:07.151ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39638,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51858 req_id = 023db374-477c-45f6-a109-7fa44999779e
57302025-04-30T06:43:07.151ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
57312025-04-30T06:43:07.151ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39638, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51858 req_id = 023db374-477c-45f6-a109-7fa44999779e
57322025-04-30T06:43:07.153ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
57332025-04-30T06:43:07.153ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
57342025-04-30T06:43:07.153ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
57352025-04-30T06:43:07.193ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f0b1eb7-3c7f-4f29-8301-392dc83ee1be", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57362025-04-30T06:43:07.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
57372025-04-30T06:43:07.221ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6430d6f-0528-4352-b8fc-c6b245ac4819", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57382025-04-30T06:43:07.222ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
57392025-04-30T06:43:07.243ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ea54831-a404-4dd3-b9c8-1700f03db7ae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57402025-04-30T06:43:07.243ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
57412025-04-30T06:43:07.246ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24c4b13e-2911-468a-a90f-e55fdf5574a3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57422025-04-30T06:43:07.246ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
57432025-04-30T06:43:07.256ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5050fc2-0916-418c-a701-d49223db4915", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57442025-04-30T06:43:07.257ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
57452025-04-30T06:43:07.262ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdc3d994-087d-4015-810a-b18bb8723da1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57462025-04-30T06:43:07.262ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
57472025-04-30T06:43:07.263ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
57482025-04-30T06:43:07.293ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ff6cef3-0af6-41b3-856f-5f05f23a27fe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57492025-04-30T06:43:07.293ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
57502025-04-30T06:43:07.295ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d778cd8b-5205-4574-bce8-6a416d6cde31", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57512025-04-30T06:43:07.295ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
57522025-04-30T06:43:07.296ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c977a60a-edba-42c9-b20e-b13529fd86bb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57532025-04-30T06:43:07.296ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
57542025-04-30T06:43:07.297ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfef4343-d927-4a31-8ccc-8269b0629001", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57552025-04-30T06:43:07.297ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
57562025-04-30T06:43:07.299ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1928c62-b33a-4483-b9b4-b5cba0722a51", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57572025-04-30T06:43:07.299ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
57582025-04-30T06:43:07.300ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3cd1d10-d312-4f43-b17f-b4cde558cc8e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57592025-04-30T06:43:07.300ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
57602025-04-30T06:43:07.301ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "345a230a-c112-489c-a644-5261808c293d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57612025-04-30T06:43:07.301ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
57622025-04-30T06:43:07.328ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57632025-04-30T06:43:07.328ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57642025-04-30T06:43:07.328ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57652025-04-30T06:43:07.328ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57662025-04-30T06:43:07.330ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57672025-04-30T06:43:07.333ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57682025-04-30T06:43:07.339ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57692025-04-30T06:43:07.339ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57702025-04-30T06:43:07.339ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57712025-04-30T06:43:07.340ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57722025-04-30T06:43:07.340ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "997a1dd9-4baa-4470-83e8-988fb4ba7d2a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57732025-04-30T06:43:07.340ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
57742025-04-30T06:43:07.343ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57752025-04-30T06:43:07.343ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57762025-04-30T06:43:07.343ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57772025-04-30T06:43:07.366ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f1ce90c-65ab-4041-8c77-01625c9c13d9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57782025-04-30T06:43:07.366ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
57792025-04-30T06:43:07.369ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "540771f5-38c3-44bf-b93e-98b42b59e664", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57802025-04-30T06:43:07.369ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
57812025-04-30T06:43:07.372ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecb64295-4617-4175-8eba-bd969935d6df", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57822025-04-30T06:43:07.372ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
57832025-04-30T06:43:07.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eb60e3a-8333-4f09-9436-e43317bfe7c0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57842025-04-30T06:43:07.380ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
57852025-04-30T06:43:07.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8f61e4a-0fe7-4448-bfe5-26e263caae19", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57862025-04-30T06:43:07.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
57872025-04-30T06:43:07.387ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62f29f21-4c4b-46d7-bb92-e6fb8e968a5b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57882025-04-30T06:43:07.388ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
57892025-04-30T06:43:07.391ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42f30bd6-d37d-443a-8271-cb1c80e05cf9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57902025-04-30T06:43:07.391ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
57912025-04-30T06:43:07.393ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2ce107d-7df4-4ea5-ac10-e3029d3e4725", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57922025-04-30T06:43:07.393ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
57932025-04-30T06:43:07.398ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ef6ed95-c123-493f-831f-019a9780fcc2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57942025-04-30T06:43:07.399ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
57952025-04-30T06:43:07.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e1d9d2a-b8a5-438d-9825-1a569fd8a26b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57962025-04-30T06:43:07.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
57972025-04-30T06:43:07.408ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
57982025-04-30T06:43:07.408ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aa4cdc7-7d6f-4189-be1a-4ec7b1076e4f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
57992025-04-30T06:43:07.408ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
58002025-04-30T06:43:07.413ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6c957e8-95f6-4c75-ac9e-6591eeae9fc1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58012025-04-30T06:43:07.413ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
58022025-04-30T06:43:07.415ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dc05be9-f7d6-4233-a012-77d872f78251", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58032025-04-30T06:43:07.415ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
58042025-04-30T06:43:07.417ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be93c334-8c6d-40ce-ac72-986f2d4f2818", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58052025-04-30T06:43:07.417ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
58062025-04-30T06:43:07.419ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf70865a-573e-448f-be50-a34d3aea14a2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58072025-04-30T06:43:07.419ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
58082025-04-30T06:43:07.422ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "363d5e97-e502-4142-9e56-bd077986aedd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58092025-04-30T06:43:07.422ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
58102025-04-30T06:43:07.423ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5523f1f6-1a01-4f62-8445-3a5c919fd107", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58112025-04-30T06:43:07.423ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
58122025-04-30T06:43:07.424ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5309856c-b1fc-40f6-bce4-5786a07dd3f6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58132025-04-30T06:43:07.425ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
58142025-04-30T06:43:07.426ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9c6ae55-d06a-4da5-9567-e8f567323cd5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58152025-04-30T06:43:07.426ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
58162025-04-30T06:43:07.428ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dad9f18f-9943-4210-8a09-38fe1e572345", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58172025-04-30T06:43:07.428ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
58182025-04-30T06:43:07.431ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf04a35c-1aa5-4fee-9838-75e0daa8f2a7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58192025-04-30T06:43:07.431ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
58202025-04-30T06:43:07.434ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db26ed26-f333-44f7-ad6f-c5a546113562", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58212025-04-30T06:43:07.434ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
58222025-04-30T06:43:07.436ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "014ec6d4-ddc4-4587-8ba7-c4d3dd4e3bc6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58232025-04-30T06:43:07.436ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
58242025-04-30T06:43:07.439ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cb81170-634b-4974-9c3d-59e6e419d539", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58252025-04-30T06:43:07.439ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
58262025-04-30T06:43:07.441ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88205017-396d-4478-a9eb-e5f803c8b5fe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58272025-04-30T06:43:07.442ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
58282025-04-30T06:43:07.447ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81f980b3-3263-4587-9454-ea94b2ab4a61", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58292025-04-30T06:43:07.447ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
58302025-04-30T06:43:07.450ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e3ba32d-41e3-4ea7-bafc-0c1c9a223da3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58312025-04-30T06:43:07.450ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
58322025-04-30T06:43:07.451ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2f03232-eea8-4c28-9450-69688b7c0e81", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58332025-04-30T06:43:07.451ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
58342025-04-30T06:43:07.452ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "514f05bc-0253-4115-9680-fdf5974a695f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58352025-04-30T06:43:07.452ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
58362025-04-30T06:43:07.453ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88e70f75-f5ed-467a-8d76-bcca0ecd1792", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58372025-04-30T06:43:07.453ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
58382025-04-30T06:43:07.455ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ab38850-2191-4d33-ab58-94304145cc17", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58392025-04-30T06:43:07.456ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
58402025-04-30T06:43:07.456ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58412025-04-30T06:43:07.456ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58422025-04-30T06:43:07.456ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58432025-04-30T06:43:07.457ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58442025-04-30T06:43:07.467ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58452025-04-30T06:43:07.467ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58462025-04-30T06:43:07.467ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58472025-04-30T06:43:07.467ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
58482025-04-30T06:43:07.468ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58492025-04-30T06:43:07.471ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58502025-04-30T06:43:07.476ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
58512025-04-30T06:43:07.476ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
58522025-04-30T06:43:07.476ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58532025-04-30T06:43:07.476ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58542025-04-30T06:43:07.476ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58552025-04-30T06:43:07.476ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58562025-04-30T06:43:07.481ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58572025-04-30T06:43:07.493ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58582025-04-30T06:43:07.493ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58592025-04-30T06:43:07.493ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
58602025-04-30T06:43:07.493ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58612025-04-30T06:43:07.495ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58622025-04-30T06:43:07.500ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
58632025-04-30T06:43:07.500ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58642025-04-30T06:43:07.500ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58652025-04-30T06:43:07.501ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58662025-04-30T06:43:07.502ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58672025-04-30T06:43:07.502ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
58682025-04-30T06:43:07.502ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58692025-04-30T06:43:07.655ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
58702025-04-30T06:43:07.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
58712025-04-30T06:43:07.656ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:55201 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
58722025-04-30T06:43:07.657ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
58732025-04-30T06:43:07.657ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58742025-04-30T06:43:07.658ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 342917a9-6156-453a-b3bf-54c5322edd4f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58752025-04-30T06:43:07.658ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:55201 req_id = 07f4c2db-f939-44c9-8f95-140fa83e9a79 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
58762025-04-30T06:43:07.658ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 342917a9-6156-453a-b3bf-54c5322edd4f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58772025-04-30T06:43:07.658ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 342917a9-6156-453a-b3bf-54c5322edd4f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58782025-04-30T06:43:07.659ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 342917a9-6156-453a-b3bf-54c5322edd4f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58792025-04-30T06:43:07.659ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 1689 local_addr = [::1]:51119 method = GET remote_addr = [::1]:55201 req_id = 07f4c2db-f939-44c9-8f95-140fa83e9a79 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
58802025-04-30T06:43:07.662ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "07f4c2db-f939-44c9-8f95-140fa83e9a79", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58812025-04-30T06:43:07.663ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
58822025-04-30T06:43:07.663ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
58832025-04-30T06:43:07.663ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
58842025-04-30T06:43:07.665ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
58852025-04-30T06:43:07.666ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58862025-04-30T06:43:07.666ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58872025-04-30T06:43:07.666ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58882025-04-30T06:43:07.667ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58892025-04-30T06:43:07.681ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 23500 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 342917a9-6156-453a-b3bf-54c5322edd4f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58902025-04-30T06:43:07.684ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "342917a9-6156-453a-b3bf-54c5322edd4f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
58912025-04-30T06:43:07.685ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
58922025-04-30T06:43:07.685ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58932025-04-30T06:43:07.691ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
58942025-04-30T06:43:07.693ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58952025-04-30T06:43:07.693ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58962025-04-30T06:43:07.694ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58972025-04-30T06:43:07.695ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2399ac5d-1b90-4648-bbdf-c2e82aace614 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
58982025-04-30T06:43:07.696ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58992025-04-30T06:43:07.696ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
59002025-04-30T06:43:07.696ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2399ac5d-1b90-4648-bbdf-c2e82aace614 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
59012025-04-30T06:43:07.696ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2399ac5d-1b90-4648-bbdf-c2e82aace614 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
59022025-04-30T06:43:07.696ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2399ac5d-1b90-4648-bbdf-c2e82aace614 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
59032025-04-30T06:43:07.696ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
59042025-04-30T06:43:07.698ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
59052025-04-30T06:43:07.702ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 6805 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2399ac5d-1b90-4648-bbdf-c2e82aace614 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
59062025-04-30T06:43:07.741ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
59072025-04-30T06:43:07.743ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
59082025-04-30T06:43:07.743ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59092025-04-30T06:43:07.743ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2399ac5d-1b90-4648-bbdf-c2e82aace614", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59102025-04-30T06:43:07.743ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
59112025-04-30T06:43:07.744ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59122025-04-30T06:43:07.744ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
59132025-04-30T06:43:07.744ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59142025-04-30T06:43:07.749ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
59152025-04-30T06:43:07.749ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
59162025-04-30T06:43:07.751ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59172025-04-30T06:43:07.754ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59182025-04-30T06:43:07.754ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59192025-04-30T06:43:07.754ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59202025-04-30T06:43:07.767ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
59212025-04-30T06:43:07.778ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
59222025-04-30T06:43:07.778ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
59232025-04-30T06:43:07.778ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 813.117764ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
59242025-04-30T06:43:07.778ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
59252025-04-30T06:43:07.778ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59262025-04-30T06:43:07.778ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59272025-04-30T06:43:07.778ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59282025-04-30T06:43:07.778ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59292025-04-30T06:43:07.779ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59302025-04-30T06:43:07.784ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59312025-04-30T06:43:07.789ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59322025-04-30T06:43:07.789ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59332025-04-30T06:43:07.789ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59342025-04-30T06:43:07.791ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59352025-04-30T06:43:07.798ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
59362025-04-30T06:43:07.799ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
59372025-04-30T06:43:07.799ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
59382025-04-30T06:43:07.800ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 63 72 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 cr..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63317 req_id = 7447b445-4466-415b-b07a-391fda12f4b8
59392025-04-30T06:43:07.800ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25458,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63317 req_id = 7447b445-4466-415b-b07a-391fda12f4b8
59402025-04-30T06:43:07.801ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
59412025-04-30T06:43:07.801ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
59422025-04-30T06:43:07.801ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63317 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7447b445-4466-415b-b07a-391fda12f4b8
59432025-04-30T06:43:07.802ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4252, tv_nsec: 601357837 } })
59442025-04-30T06:43:07.802ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
59452025-04-30T06:43:07.802ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ca 71 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .q..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56315 req_id = 06d43ece-8b73-43e4-bd53-a01a22c7051c
59462025-04-30T06:43:07.803ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51825,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56315 req_id = 06d43ece-8b73-43e4-bd53-a01a22c7051c
59472025-04-30T06:43:07.803ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
59482025-04-30T06:43:07.803ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56315 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 06d43ece-8b73-43e4-bd53-a01a22c7051c
59492025-04-30T06:43:07.869ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
59502025-04-30T06:43:07.870ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59512025-04-30T06:43:07.871ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59522025-04-30T06:43:07.871ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59532025-04-30T06:43:07.871ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
59542025-04-30T06:43:07.872ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59552025-04-30T06:43:07.875ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 25 1f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c %............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62038 req_id = 25342991-5474-4aa2-9254-56809a536421
59562025-04-30T06:43:07.875ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9503,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62038 req_id = 25342991-5474-4aa2-9254-56809a536421
59572025-04-30T06:43:07.875ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
59582025-04-30T06:43:07.875ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9503, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62038 req_id = 25342991-5474-4aa2-9254-56809a536421
59592025-04-30T06:43:07.882ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
59602025-04-30T06:43:07.882ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
59612025-04-30T06:43:07.882ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
59622025-04-30T06:43:07.890ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59632025-04-30T06:43:07.902ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59642025-04-30T06:43:07.902ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59652025-04-30T06:43:07.902ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
59662025-04-30T06:43:07.903ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59672025-04-30T06:43:07.909ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59682025-04-30T06:43:07.909ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
59692025-04-30T06:43:07.909ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59702025-04-30T06:43:07.909ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8693de5-63eb-4648-b4e4-61d0bc37260d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59712025-04-30T06:43:07.910ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
59722025-04-30T06:43:07.913ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f803b5d-f4d2-4e61-8dba-8eff891927e1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59732025-04-30T06:43:07.914ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
59742025-04-30T06:43:07.917ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75765bd1-9a59-42ed-9f0b-c8e262626a5f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59752025-04-30T06:43:07.917ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
59762025-04-30T06:43:07.919ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "047d6d8a-f2cc-4d91-9efd-00dde14d7588", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59772025-04-30T06:43:07.919ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
59782025-04-30T06:43:07.922ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb4f8f49-476f-477b-9a4c-376808a30eb8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59792025-04-30T06:43:07.922ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
59802025-04-30T06:43:07.925ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1136706e-2b7e-4ef8-abef-8f90d43cca26", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59812025-04-30T06:43:07.925ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
59822025-04-30T06:43:07.927ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fc33fb8-58a0-498a-85f8-d24d001ecac5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59832025-04-30T06:43:07.928ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
59842025-04-30T06:43:07.931ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e84c504-e615-43d8-a2ec-45d596f4d9d9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59852025-04-30T06:43:07.932ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
59862025-04-30T06:43:07.932ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "208b29d2-9444-4b64-8154-58e69e0bc4b2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59872025-04-30T06:43:07.933ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
59882025-04-30T06:43:07.934ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9821ded8-16ed-4eff-9dd4-fa0f9a70eca1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59892025-04-30T06:43:07.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
59902025-04-30T06:43:07.936ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0166787-9f5e-4b14-8d7a-435cee6cc573", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59912025-04-30T06:43:07.936ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
59922025-04-30T06:43:07.941ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bc6da26-2df5-415b-b48a-61f634d81d2f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59932025-04-30T06:43:07.941ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
59942025-04-30T06:43:07.942ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
59952025-04-30T06:43:07.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "386186c9-85fc-4eea-890d-c283a62f888e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59962025-04-30T06:43:07.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
59972025-04-30T06:43:07.949ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f256085-0693-4e66-b483-4ea00bcc873d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
59982025-04-30T06:43:07.950ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
59992025-04-30T06:43:07.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b73299e-7d45-4a3a-b6e2-8e9e34d48d42", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60002025-04-30T06:43:07.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
60012025-04-30T06:43:07.957ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6a3a218-2631-40ac-9553-e4187c495797", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60022025-04-30T06:43:07.957ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
60032025-04-30T06:43:07.961ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f552e0bf-b76b-4d74-9fac-b47d7ef323dd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60042025-04-30T06:43:07.961ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
60052025-04-30T06:43:07.969ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54f051e0-f9ba-4c8a-84c0-f3f6361aa07b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60062025-04-30T06:43:07.975ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
60072025-04-30T06:43:07.977ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "331efde5-ff06-4f39-ae2f-77bd33d5a8e6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60082025-04-30T06:43:07.977ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
60092025-04-30T06:43:07.981ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c4479eb-0d8b-4042-aac8-e7764e8fe125", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60102025-04-30T06:43:07.981ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
60112025-04-30T06:43:07.987ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "356c9b2c-66fa-4e88-aff6-6c53d1120018", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60122025-04-30T06:43:07.987ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
60132025-04-30T06:43:07.993ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60142025-04-30T06:43:07.993ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60152025-04-30T06:43:07.994ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60162025-04-30T06:43:07.994ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60172025-04-30T06:43:07.994ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60182025-04-30T06:43:07.999ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25f1a7fe-79ed-4350-b391-f8179879fa79", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60192025-04-30T06:43:07.999ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
60202025-04-30T06:43:07.999ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60212025-04-30T06:43:08.011ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60222025-04-30T06:43:08.011ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60232025-04-30T06:43:08.011ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60242025-04-30T06:43:08.011ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60252025-04-30T06:43:08.028ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60262025-04-30T06:43:08.028ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60272025-04-30T06:43:08.029ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60282025-04-30T06:43:08.030ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3080acb9-0d02-409c-842b-d4f4e249cf0b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60292025-04-30T06:43:08.030ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
60302025-04-30T06:43:08.047ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68a9d410-4c72-42e0-ab21-2f6e4e7ab82c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60312025-04-30T06:43:08.047ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
60322025-04-30T06:43:08.050ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c07b0f0d-a9fd-46ec-a11b-a7df64c0005e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60332025-04-30T06:43:08.050ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
60342025-04-30T06:43:08.052ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6e015fa-e6bd-44b3-b5b2-b809b644205f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60352025-04-30T06:43:08.052ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
60362025-04-30T06:43:08.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e2dd8d1-5670-4909-88d7-871ac77847de", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60372025-04-30T06:43:08.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
60382025-04-30T06:43:08.058ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07e99861-d26e-42e9-946c-47d9272078c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60392025-04-30T06:43:08.058ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
60402025-04-30T06:43:08.063ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d682973e-8046-44fd-ab6d-9d41e890f8c1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60412025-04-30T06:43:08.063ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
60422025-04-30T06:43:08.067ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "298a989c-aab9-4968-8c70-be8b1d5bf85c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60432025-04-30T06:43:08.067ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
60442025-04-30T06:43:08.074ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
60452025-04-30T06:43:08.074ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ead08899-8dc6-4374-8576-a027c9abcf6a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60462025-04-30T06:43:08.075ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
60472025-04-30T06:43:08.078ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "021951a8-53d8-497e-85a1-41e4d354ba81", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60482025-04-30T06:43:08.078ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
60492025-04-30T06:43:08.079ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c7c8820-3672-4cff-a0cf-73ef57d09422", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60502025-04-30T06:43:08.080ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
60512025-04-30T06:43:08.081ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d13aa2d5-0a27-4961-ad70-6b762eb1c3b4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60522025-04-30T06:43:08.081ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
60532025-04-30T06:43:08.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66cbf034-4da2-483f-96c7-022d22a7df30", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60542025-04-30T06:43:08.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
60552025-04-30T06:43:08.084ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "487b0e7d-aae6-41ff-a2e0-3495c1c3c04e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60562025-04-30T06:43:08.084ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
60572025-04-30T06:43:08.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7a04eff-8683-4041-80d7-3097610657bf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60582025-04-30T06:43:08.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
60592025-04-30T06:43:08.088ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec6321c4-8ae2-493d-8f1a-68d5d4ac705c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60602025-04-30T06:43:08.088ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
60612025-04-30T06:43:08.093ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99c0a587-8918-4154-979b-faf3bee6c9f1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60622025-04-30T06:43:08.093ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
60632025-04-30T06:43:08.096ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed17e83-8430-4688-8dc2-fd81966f8812", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60642025-04-30T06:43:08.097ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
60652025-04-30T06:43:08.100ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbd951e3-b875-4cf9-ae2b-ecebe108187d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60662025-04-30T06:43:08.100ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
60672025-04-30T06:43:08.101ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56c25dd7-ef77-4454-bf71-0b4ee3f47e96", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60682025-04-30T06:43:08.102ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
60692025-04-30T06:43:08.103ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a295c78-50eb-49b9-899d-f95c8bee3e6c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60702025-04-30T06:43:08.103ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
60712025-04-30T06:43:08.104ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb09810f-912b-4a0c-86bf-1ffce7d2e1e7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60722025-04-30T06:43:08.105ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
60732025-04-30T06:43:08.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d953466-72f0-48cd-973f-4d60d0193eb8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
60742025-04-30T06:43:08.106ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
60752025-04-30T06:43:08.106ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60762025-04-30T06:43:08.106ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60772025-04-30T06:43:08.106ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60782025-04-30T06:43:08.108ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60792025-04-30T06:43:08.114ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60802025-04-30T06:43:08.119ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
60812025-04-30T06:43:08.119ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
60822025-04-30T06:43:08.119ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60832025-04-30T06:43:08.119ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60842025-04-30T06:43:08.119ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60852025-04-30T06:43:08.120ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60862025-04-30T06:43:08.123ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60872025-04-30T06:43:08.129ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
60882025-04-30T06:43:08.129ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60892025-04-30T06:43:08.129ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60902025-04-30T06:43:08.129ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60912025-04-30T06:43:08.130ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60922025-04-30T06:43:08.130ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60932025-04-30T06:43:08.130ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60942025-04-30T06:43:08.130ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60952025-04-30T06:43:08.131ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60962025-04-30T06:43:08.134ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60972025-04-30T06:43:08.140ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60982025-04-30T06:43:08.140ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60992025-04-30T06:43:08.140ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
61002025-04-30T06:43:08.140ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61012025-04-30T06:43:08.194ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
61022025-04-30T06:43:08.194ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
61032025-04-30T06:43:08.194ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:33381 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
61042025-04-30T06:43:08.196ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61052025-04-30T06:43:08.196ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
61062025-04-30T06:43:08.197ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61072025-04-30T06:43:08.197ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:33381 req_id = 173605e7-3e04-45d0-b51b-1aa15db98b4f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
61082025-04-30T06:43:08.197ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 545 local_addr = [::1]:51119 method = GET remote_addr = [::1]:33381 req_id = 173605e7-3e04-45d0-b51b-1aa15db98b4f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
61092025-04-30T06:43:08.198ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "173605e7-3e04-45d0-b51b-1aa15db98b4f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61102025-04-30T06:43:08.201ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
61112025-04-30T06:43:08.201ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
61122025-04-30T06:43:08.201ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
61132025-04-30T06:43:08.204ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
61142025-04-30T06:43:08.204ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61152025-04-30T06:43:08.204ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61162025-04-30T06:43:08.204ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61172025-04-30T06:43:08.204ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61182025-04-30T06:43:08.208ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61192025-04-30T06:43:08.212ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
61202025-04-30T06:43:08.212ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61212025-04-30T06:43:08.212ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61222025-04-30T06:43:08.212ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61232025-04-30T06:43:08.300ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
61242025-04-30T06:43:08.302ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
61252025-04-30T06:43:08.302ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
61262025-04-30T06:43:08.302ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 523.755067ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
61272025-04-30T06:43:08.302ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
61282025-04-30T06:43:08.302ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
61292025-04-30T06:43:08.303ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61302025-04-30T06:43:08.304ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61312025-04-30T06:43:08.304ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61322025-04-30T06:43:08.305ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61332025-04-30T06:43:08.309ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61342025-04-30T06:43:08.313ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
61352025-04-30T06:43:08.313ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61362025-04-30T06:43:08.313ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61372025-04-30T06:43:08.314ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61382025-04-30T06:43:08.318ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
61392025-04-30T06:43:08.318ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
61402025-04-30T06:43:08.318ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0c 3d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .=..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53227 req_id = ca466166-2bf6-4536-a857-ba305b06f149
61412025-04-30T06:43:08.319ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3133,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53227 req_id = ca466166-2bf6-4536-a857-ba305b06f149
61422025-04-30T06:43:08.319ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
61432025-04-30T06:43:08.319ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
61442025-04-30T06:43:08.320ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53227 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ca466166-2bf6-4536-a857-ba305b06f149
61452025-04-30T06:43:08.320ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4253, tv_nsec: 119763547 } })
61462025-04-30T06:43:08.320ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
61472025-04-30T06:43:08.321ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0c bd 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63246 req_id = 31a454f8-fef7-44ad-b436-650cdaeaa547
61482025-04-30T06:43:08.321ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3261,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63246 req_id = 31a454f8-fef7-44ad-b436-650cdaeaa547
61492025-04-30T06:43:08.321ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
61502025-04-30T06:43:08.321ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63246 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 31a454f8-fef7-44ad-b436-650cdaeaa547
61512025-04-30T06:43:08.368ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
61522025-04-30T06:43:08.369ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61532025-04-30T06:43:08.369ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61542025-04-30T06:43:08.369ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61552025-04-30T06:43:08.369ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
61562025-04-30T06:43:08.370ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61572025-04-30T06:43:08.370ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c2 04 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56867 req_id = dfcd3c0c-f20d-4d80-aec0-4ff414e5910d
61582025-04-30T06:43:08.371ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49668,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56867 req_id = dfcd3c0c-f20d-4d80-aec0-4ff414e5910d
61592025-04-30T06:43:08.371ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
61602025-04-30T06:43:08.371ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49668, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56867 req_id = dfcd3c0c-f20d-4d80-aec0-4ff414e5910d
61612025-04-30T06:43:08.371ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
61622025-04-30T06:43:08.371ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
61632025-04-30T06:43:08.371ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
61642025-04-30T06:43:08.373ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61652025-04-30T06:43:08.378ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61662025-04-30T06:43:08.378ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61672025-04-30T06:43:08.379ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
61682025-04-30T06:43:08.380ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61692025-04-30T06:43:08.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "91b76e49-91fd-4a52-b912-10a424d6a7a9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61702025-04-30T06:43:08.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
61712025-04-30T06:43:08.383ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61722025-04-30T06:43:08.383ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
61732025-04-30T06:43:08.384ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61742025-04-30T06:43:08.388ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4992e005-6a59-4271-b27a-da0c7aec1307", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61752025-04-30T06:43:08.389ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
61762025-04-30T06:43:08.394ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49bc1fd7-a4c6-4c48-95bf-247ac8e58ec2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61772025-04-30T06:43:08.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
61782025-04-30T06:43:08.400ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8adbaf31-ac7e-48f3-8251-a4815c2db629", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61792025-04-30T06:43:08.400ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
61802025-04-30T06:43:08.406ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c71a844-3edb-462f-9690-9a9a87fdb0dc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61812025-04-30T06:43:08.407ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
61822025-04-30T06:43:08.412ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3db7f1b0-b2b0-4db7-8126-f1eedfbb5f13", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61832025-04-30T06:43:08.413ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
61842025-04-30T06:43:08.417ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07eefdca-ede4-4190-bc91-727fa45c215c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61852025-04-30T06:43:08.418ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
61862025-04-30T06:43:08.425ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
61872025-04-30T06:43:08.429ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8b724f8-9797-4b93-a77c-554897fa186b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61882025-04-30T06:43:08.430ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
61892025-04-30T06:43:08.431ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff57853-e370-4745-bee1-e6c76ce07a11", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61902025-04-30T06:43:08.432ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
61912025-04-30T06:43:08.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "104bec67-6170-4d74-b73c-f0ddc3b7d7da", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61922025-04-30T06:43:08.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
61932025-04-30T06:43:08.436ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a62e2dfa-698b-464d-a92c-439c5c52dda8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61942025-04-30T06:43:08.436ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
61952025-04-30T06:43:08.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "742d7f8c-3c89-4bbe-b111-aa343514744e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61962025-04-30T06:43:08.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
61972025-04-30T06:43:08.440ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db1bca76-22ed-471e-94bf-5d3a6f92b81b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
61982025-04-30T06:43:08.440ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
61992025-04-30T06:43:08.444ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "924ab155-ec4a-4f75-b9a3-603deb16b557", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62002025-04-30T06:43:08.444ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
62012025-04-30T06:43:08.446ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb90cf61-2811-4c7e-af87-90ea753c309f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62022025-04-30T06:43:08.446ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
62032025-04-30T06:43:08.452ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d45bf7fe-73bf-4cbc-aafe-c2f4129e33aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62042025-04-30T06:43:08.452ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
62052025-04-30T06:43:08.456ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a602305-978d-4892-a925-1991406c2621", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62062025-04-30T06:43:08.456ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
62072025-04-30T06:43:08.461ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7b7037c-bde3-4f86-a3a5-09ddbe714210", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62082025-04-30T06:43:08.462ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
62092025-04-30T06:43:08.477ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b562f8be-c27f-4fee-aac1-d11ed245bb05", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62102025-04-30T06:43:08.477ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
62112025-04-30T06:43:08.478ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62122025-04-30T06:43:08.478ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62132025-04-30T06:43:08.478ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62142025-04-30T06:43:08.478ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
62152025-04-30T06:43:08.478ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62162025-04-30T06:43:08.481ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e81aeb18-6361-4857-91af-15a0c695f0f8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62172025-04-30T06:43:08.482ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
62182025-04-30T06:43:08.482ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62192025-04-30T06:43:08.487ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62202025-04-30T06:43:08.488ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62212025-04-30T06:43:08.488ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
62222025-04-30T06:43:08.488ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62232025-04-30T06:43:08.490ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f496e8c0-a151-4e05-8746-b2ed92bea7e7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62242025-04-30T06:43:08.490ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
62252025-04-30T06:43:08.491ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62262025-04-30T06:43:08.491ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
62272025-04-30T06:43:08.491ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62282025-04-30T06:43:08.492ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33ed15b3-3eec-42a4-8f6e-6371e694ab36", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62292025-04-30T06:43:08.492ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
62302025-04-30T06:43:08.494ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71fa0ef7-f5ff-420c-8230-d4fd13059c31", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62312025-04-30T06:43:08.495ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
62322025-04-30T06:43:08.499ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eceb1b43-4dc5-4905-904e-8b07a0a1372f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62332025-04-30T06:43:08.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
62342025-04-30T06:43:08.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8397d752-89d9-416b-a857-bdd22347a5b6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62352025-04-30T06:43:08.506ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
62362025-04-30T06:43:08.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "585b47aa-58ec-4a16-9993-ac9d820e9f43", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62372025-04-30T06:43:08.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
62382025-04-30T06:43:08.518ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5001da06-3363-4c7b-9394-861b57124f6a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62392025-04-30T06:43:08.519ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
62402025-04-30T06:43:08.522ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dc3fac8-334e-4cd8-8460-a15307852b07", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62412025-04-30T06:43:08.522ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
62422025-04-30T06:43:08.524ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87fb9e42-847e-4b90-8539-833d1a675bb1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62432025-04-30T06:43:08.524ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
62442025-04-30T06:43:08.526ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e54173ff-b980-4768-b102-83e5490afaa3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62452025-04-30T06:43:08.526ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
62462025-04-30T06:43:08.527ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d5fe9d7-32f3-43e3-b1aa-690ab4b700b5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62472025-04-30T06:43:08.527ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
62482025-04-30T06:43:08.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed92af2d-9cba-4fca-a19d-e6ea1e951700", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62492025-04-30T06:43:08.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
62502025-04-30T06:43:08.531ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
62512025-04-30T06:43:08.532ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ba63b86-7ba6-4a0a-8efb-fbf03cc14566", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62522025-04-30T06:43:08.532ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
62532025-04-30T06:43:08.533ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4ebacbf-dd99-4c58-9be4-50212651f9cb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62542025-04-30T06:43:08.533ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
62552025-04-30T06:43:08.534ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "723f9426-f7a8-407f-9f80-107b6c6ef758", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62562025-04-30T06:43:08.534ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
62572025-04-30T06:43:08.536ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ee3f90e-8555-45e0-9e80-12fd2bf67681", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62582025-04-30T06:43:08.536ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
62592025-04-30T06:43:08.539ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "065ff8e5-a4c9-4a22-b670-a8de7e932bf3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62602025-04-30T06:43:08.539ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
62612025-04-30T06:43:08.541ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2224e3f3-e8f4-4cf0-bba8-9ee00843814f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62622025-04-30T06:43:08.541ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
62632025-04-30T06:43:08.544ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95fe46ce-7dc5-43c2-841c-00fa8b541a11", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62642025-04-30T06:43:08.544ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
62652025-04-30T06:43:08.547ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3898fc2-d552-4cef-be35-e836892d5cfb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62662025-04-30T06:43:08.547ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
62672025-04-30T06:43:08.550ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4346206a-a8aa-48a0-b69b-979e8e418fc2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62682025-04-30T06:43:08.550ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
62692025-04-30T06:43:08.551ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e3610a1-183f-4cb9-8fdd-9755c64b0933", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62702025-04-30T06:43:08.551ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
62712025-04-30T06:43:08.552ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4d156f5-626c-4996-8ebe-1c549f52aa12", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62722025-04-30T06:43:08.552ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
62732025-04-30T06:43:08.554ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7620dffe-1f79-4e8f-b2c0-0583e584d562", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62742025-04-30T06:43:08.554ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
62752025-04-30T06:43:08.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "758e7488-33c4-41f1-8763-c6db1a451e3e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
62762025-04-30T06:43:08.557ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
62772025-04-30T06:43:08.557ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62782025-04-30T06:43:08.557ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62792025-04-30T06:43:08.557ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62802025-04-30T06:43:08.558ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62812025-04-30T06:43:08.573ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62822025-04-30T06:43:08.593ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
62832025-04-30T06:43:08.593ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
62842025-04-30T06:43:08.595ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62852025-04-30T06:43:08.595ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62862025-04-30T06:43:08.595ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62872025-04-30T06:43:08.596ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62882025-04-30T06:43:08.597ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62892025-04-30T06:43:08.597ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62902025-04-30T06:43:08.597ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62912025-04-30T06:43:08.597ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
62922025-04-30T06:43:08.598ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62932025-04-30T06:43:08.601ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62942025-04-30T06:43:08.607ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
62952025-04-30T06:43:08.607ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62962025-04-30T06:43:08.607ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62972025-04-30T06:43:08.608ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62982025-04-30T06:43:08.610ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62992025-04-30T06:43:08.616ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63002025-04-30T06:43:08.616ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63012025-04-30T06:43:08.616ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
63022025-04-30T06:43:08.617ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63032025-04-30T06:43:08.674ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
63042025-04-30T06:43:08.674ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
63052025-04-30T06:43:08.675ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:47262 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
63062025-04-30T06:43:08.676ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:47262 req_id = 376cd2e1-2a6f-4169-b3cb-d9f2bae45c40 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
63072025-04-30T06:43:08.676ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 607 local_addr = [::1]:51119 method = GET remote_addr = [::1]:47262 req_id = 376cd2e1-2a6f-4169-b3cb-d9f2bae45c40 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
63082025-04-30T06:43:08.677ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63092025-04-30T06:43:08.677ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
63102025-04-30T06:43:08.678ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63112025-04-30T06:43:08.678ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "376cd2e1-2a6f-4169-b3cb-d9f2bae45c40", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
63122025-04-30T06:43:08.681ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
63132025-04-30T06:43:08.681ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
63142025-04-30T06:43:08.681ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
63152025-04-30T06:43:08.683ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
63162025-04-30T06:43:08.683ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63172025-04-30T06:43:08.683ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63182025-04-30T06:43:08.683ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63192025-04-30T06:43:08.684ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63202025-04-30T06:43:08.688ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
63212025-04-30T06:43:08.688ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
63222025-04-30T06:43:08.688ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
63232025-04-30T06:43:08.689ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
63242025-04-30T06:43:08.690ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63252025-04-30T06:43:08.695ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
63262025-04-30T06:43:08.695ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63272025-04-30T06:43:08.695ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63282025-04-30T06:43:08.696ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63292025-04-30T06:43:08.700ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
63302025-04-30T06:43:08.701ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
63312025-04-30T06:43:08.705ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
63322025-04-30T06:43:08.705ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
63332025-04-30T06:43:08.732ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
63342025-04-30T06:43:08.732ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
63352025-04-30T06:43:08.733ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
63362025-04-30T06:43:08.735ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
63372025-04-30T06:43:08.737ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
63382025-04-30T06:43:08.737ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
63392025-04-30T06:43:08.737ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
63402025-04-30T06:43:08.741ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
63412025-04-30T06:43:08.741ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
63422025-04-30T06:43:08.741ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 439.225238ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
63432025-04-30T06:43:08.742ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
63442025-04-30T06:43:08.742ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
63452025-04-30T06:43:08.742ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63462025-04-30T06:43:08.742ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63472025-04-30T06:43:08.742ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63482025-04-30T06:43:08.743ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63492025-04-30T06:43:08.757ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63502025-04-30T06:43:08.763ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
63512025-04-30T06:43:08.763ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63522025-04-30T06:43:08.763ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63532025-04-30T06:43:08.764ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63542025-04-30T06:43:08.773ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
63552025-04-30T06:43:08.773ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
63562025-04-30T06:43:08.776ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4c f1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 L............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59535 req_id = a71e5591-dfb8-4154-bf4f-80c3fb519178
63572025-04-30T06:43:08.776ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19697,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59535 req_id = a71e5591-dfb8-4154-bf4f-80c3fb519178
63582025-04-30T06:43:08.776ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
63592025-04-30T06:43:08.776ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
63602025-04-30T06:43:08.777ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59535 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a71e5591-dfb8-4154-bf4f-80c3fb519178
63612025-04-30T06:43:08.778ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4253, tv_nsec: 577021413 } })
63622025-04-30T06:43:08.778ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
63632025-04-30T06:43:08.778ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6e 13 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e n...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62065 req_id = f844b888-6e3f-4260-8482-a88cd904f3e7
63642025-04-30T06:43:08.778ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28179,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62065 req_id = f844b888-6e3f-4260-8482-a88cd904f3e7
63652025-04-30T06:43:08.778ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
63662025-04-30T06:43:08.778ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62065 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f844b888-6e3f-4260-8482-a88cd904f3e7
63672025-04-30T06:43:08.837ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
63682025-04-30T06:43:08.838ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63692025-04-30T06:43:08.838ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63702025-04-30T06:43:08.838ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63712025-04-30T06:43:08.838ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
63722025-04-30T06:43:08.839ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63732025-04-30T06:43:08.840ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
63742025-04-30T06:43:08.841ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e4 aa 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65338 req_id = 9ae01cf7-20d7-43ed-9cfb-fa8a866584ea
63752025-04-30T06:43:08.841ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58538,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65338 req_id = 9ae01cf7-20d7-43ed-9cfb-fa8a866584ea
63762025-04-30T06:43:08.842ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
63772025-04-30T06:43:08.842ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58538, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65338 req_id = 9ae01cf7-20d7-43ed-9cfb-fa8a866584ea
63782025-04-30T06:43:08.843ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
63792025-04-30T06:43:08.843ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
63802025-04-30T06:43:08.843ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
63812025-04-30T06:43:08.845ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63822025-04-30T06:43:08.851ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63832025-04-30T06:43:08.851ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63842025-04-30T06:43:08.851ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
63852025-04-30T06:43:08.851ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63862025-04-30T06:43:08.854ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d704b6e-04df-4396-abd3-de2e4daf7d88", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
63872025-04-30T06:43:08.854ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
63882025-04-30T06:43:08.855ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63892025-04-30T06:43:08.855ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
63902025-04-30T06:43:08.855ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63912025-04-30T06:43:08.857ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c28058-a91d-45ae-b10f-b56227760e28", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
63922025-04-30T06:43:08.857ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
63932025-04-30T06:43:08.862ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "124f010d-1914-436f-a486-f59c99e91a59", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
63942025-04-30T06:43:08.862ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
63952025-04-30T06:43:08.866ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a294730-039d-4c3e-8f3e-810bd08699dd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
63962025-04-30T06:43:08.866ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
63972025-04-30T06:43:08.870ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee377963-d6a9-4c54-b78a-85dfc89d0e2d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
63982025-04-30T06:43:08.870ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
63992025-04-30T06:43:08.872ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6d682a4-da4f-4448-9f01-a3b9cb033a9a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64002025-04-30T06:43:08.872ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
64012025-04-30T06:43:08.878ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cf5b4d6-b0aa-4897-803d-04753c9874ed", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64022025-04-30T06:43:08.878ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
64032025-04-30T06:43:08.881ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7012c4a-0cb2-4040-9479-74278d0dc5bb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64042025-04-30T06:43:08.882ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
64052025-04-30T06:43:08.883ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fbcef08-dd13-4ca4-9c60-fb18372fee9e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64062025-04-30T06:43:08.883ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
64072025-04-30T06:43:08.886ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8668fb7b-b6de-400e-8170-395f15627b5b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64082025-04-30T06:43:08.887ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
64092025-04-30T06:43:08.887ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
64102025-04-30T06:43:08.890ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbfd516e-9b4d-4349-9821-3585423f6943", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64112025-04-30T06:43:08.891ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
64122025-04-30T06:43:08.900ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "580c10fe-547e-4077-b65a-c7e0ac490cdc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64132025-04-30T06:43:08.901ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
64142025-04-30T06:43:08.902ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ad695cc-446b-4518-bdbd-7aee028b1146", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64152025-04-30T06:43:08.903ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
64162025-04-30T06:43:08.907ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c4932a5-b1ee-4aa4-a3f9-bf7762ca417a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64172025-04-30T06:43:08.907ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
64182025-04-30T06:43:08.909ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea908568-5c74-4548-ac9f-a9a67d9ae399", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64192025-04-30T06:43:08.910ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
64202025-04-30T06:43:08.912ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a44086b3-ad23-4ed6-9350-d09e0f2a0d6f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64212025-04-30T06:43:08.912ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
64222025-04-30T06:43:08.916ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8248ac55-0db3-4998-bf2f-e05bb853fab2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64232025-04-30T06:43:08.916ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
64242025-04-30T06:43:08.918ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a686b52a-7097-44be-996d-4a858595a22a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64252025-04-30T06:43:08.918ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
64262025-04-30T06:43:08.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6c6444e-5977-4619-9aca-9f1facb1c095", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64272025-04-30T06:43:08.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
64282025-04-30T06:43:08.921ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2edcfb3-52be-46eb-b1b2-45c890f6c343", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64292025-04-30T06:43:08.921ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
64302025-04-30T06:43:08.922ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da1abeff-e88f-4a2c-b4c2-49a727058a21", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64312025-04-30T06:43:08.922ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
64322025-04-30T06:43:08.923ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "396635b0-1d71-4f47-900d-a11a5fbccf10", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64332025-04-30T06:43:08.923ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
64342025-04-30T06:43:08.924ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c00bcd64-60da-478e-b89a-adf8befd8be4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64352025-04-30T06:43:08.924ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
64362025-04-30T06:43:08.926ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b62dc139-44ef-46ce-8b82-fcc828d62082", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64372025-04-30T06:43:08.926ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
64382025-04-30T06:43:08.928ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f59698ff-b41e-49db-b6df-bb1505b6b9a9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64392025-04-30T06:43:08.928ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
64402025-04-30T06:43:08.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34da6b05-f937-4d78-ba90-c922a5549b17", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64412025-04-30T06:43:08.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
64422025-04-30T06:43:08.938ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d67a31e-4ec2-465c-8169-d6a72b1ee1c0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64432025-04-30T06:43:08.938ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
64442025-04-30T06:43:08.938ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64452025-04-30T06:43:08.938ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64462025-04-30T06:43:08.938ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64472025-04-30T06:43:08.939ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
64482025-04-30T06:43:08.939ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64492025-04-30T06:43:08.942ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d444f07-bef2-4263-9d23-047f036bd46b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64502025-04-30T06:43:08.942ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
64512025-04-30T06:43:08.942ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64522025-04-30T06:43:08.951ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64532025-04-30T06:43:08.951ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64542025-04-30T06:43:08.951ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
64552025-04-30T06:43:08.952ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64562025-04-30T06:43:08.957ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64572025-04-30T06:43:08.957ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
64582025-04-30T06:43:08.958ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64592025-04-30T06:43:08.968ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58041226-ff71-4850-a4a8-dce91b9f375e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64602025-04-30T06:43:08.968ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
64612025-04-30T06:43:08.972ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c08cd8d6-14d3-4e0b-8496-e19f7a941ac2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64622025-04-30T06:43:08.972ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
64632025-04-30T06:43:08.985ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f47dbe3-caa9-4841-9c01-b699d74e6296", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64642025-04-30T06:43:08.986ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
64652025-04-30T06:43:09.003ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c159f09-0ba4-4608-95c1-d0bec9c4a808", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64662025-04-30T06:43:09.003ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
64672025-04-30T06:43:09.015ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b51314e-dab7-4b74-9b0b-acab10d2c647", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64682025-04-30T06:43:09.015ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
64692025-04-30T06:43:09.020ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
64702025-04-30T06:43:09.039ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d980014a-52df-415b-9316-738ee2221caa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64712025-04-30T06:43:09.039ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
64722025-04-30T06:43:09.044ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e34c3c3-8b33-4871-b19c-7f9b2c5f9935", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64732025-04-30T06:43:09.044ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
64742025-04-30T06:43:09.046ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c11f63f-e30e-45ad-b367-fa849bc4647c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64752025-04-30T06:43:09.046ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
64762025-04-30T06:43:09.048ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2060023c-8fda-4a8c-8f16-864f27f1d0ce", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64772025-04-30T06:43:09.048ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
64782025-04-30T06:43:09.050ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31a40bcf-7999-4f33-961d-51eb956b3e90", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64792025-04-30T06:43:09.050ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
64802025-04-30T06:43:09.052ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e2b51ef-74f9-4dea-89c6-fdf0c92d57fe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64812025-04-30T06:43:09.052ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
64822025-04-30T06:43:09.056ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58bae7f1-4041-4462-96d5-fbe7b9900a09", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64832025-04-30T06:43:09.056ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
64842025-04-30T06:43:09.059ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35856b48-2f5b-4f46-8c8e-63c0f66ff81d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64852025-04-30T06:43:09.059ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
64862025-04-30T06:43:09.060ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfc25d76-9d17-4516-b9ce-df741557672b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64872025-04-30T06:43:09.060ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
64882025-04-30T06:43:09.062ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8057f53-0904-4e38-991a-34875a9782c3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64892025-04-30T06:43:09.062ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
64902025-04-30T06:43:09.063ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "510f4dde-4672-40e9-bb6e-b5b78a933ce8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64912025-04-30T06:43:09.064ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
64922025-04-30T06:43:09.067ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae498062-9506-4774-a93e-7cdd17a14f5b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
64932025-04-30T06:43:09.067ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
64942025-04-30T06:43:09.067ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64952025-04-30T06:43:09.068ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64962025-04-30T06:43:09.068ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64972025-04-30T06:43:09.069ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64982025-04-30T06:43:09.079ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64992025-04-30T06:43:09.079ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65002025-04-30T06:43:09.079ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65012025-04-30T06:43:09.079ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65022025-04-30T06:43:09.080ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65032025-04-30T06:43:09.082ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65042025-04-30T06:43:09.091ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65052025-04-30T06:43:09.091ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65062025-04-30T06:43:09.091ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65072025-04-30T06:43:09.091ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65082025-04-30T06:43:09.091ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65092025-04-30T06:43:09.092ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65102025-04-30T06:43:09.094ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65112025-04-30T06:43:09.100ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65122025-04-30T06:43:09.100ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65132025-04-30T06:43:09.100ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65142025-04-30T06:43:09.100ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65152025-04-30T06:43:09.103ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65162025-04-30T06:43:09.108ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65172025-04-30T06:43:09.108ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65182025-04-30T06:43:09.108ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65192025-04-30T06:43:09.108ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65202025-04-30T06:43:09.109ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65212025-04-30T06:43:09.109ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65222025-04-30T06:43:09.111ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65232025-04-30T06:43:09.165ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
65242025-04-30T06:43:09.165ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
65252025-04-30T06:43:09.166ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:38185 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
65262025-04-30T06:43:09.167ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:38185 req_id = caac58a9-47f9-4f91-afe5-983e09d0412a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65272025-04-30T06:43:09.168ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 634 local_addr = [::1]:51119 method = GET remote_addr = [::1]:38185 req_id = caac58a9-47f9-4f91-afe5-983e09d0412a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65282025-04-30T06:43:09.168ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "caac58a9-47f9-4f91-afe5-983e09d0412a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
65292025-04-30T06:43:09.170ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65302025-04-30T06:43:09.170ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65312025-04-30T06:43:09.170ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
65322025-04-30T06:43:09.172ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
65332025-04-30T06:43:09.172ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65342025-04-30T06:43:09.172ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65352025-04-30T06:43:09.172ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65362025-04-30T06:43:09.172ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65372025-04-30T06:43:09.175ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65382025-04-30T06:43:09.179ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
65392025-04-30T06:43:09.180ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65402025-04-30T06:43:09.180ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65412025-04-30T06:43:09.180ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65422025-04-30T06:43:09.238ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
65432025-04-30T06:43:09.252ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
65442025-04-30T06:43:09.252ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
65452025-04-30T06:43:09.253ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 511.022278ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
65462025-04-30T06:43:09.253ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
65472025-04-30T06:43:09.253ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
65482025-04-30T06:43:09.253ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65492025-04-30T06:43:09.253ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65502025-04-30T06:43:09.253ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65512025-04-30T06:43:09.255ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65522025-04-30T06:43:09.258ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65532025-04-30T06:43:09.262ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
65542025-04-30T06:43:09.263ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65552025-04-30T06:43:09.263ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65562025-04-30T06:43:09.263ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65572025-04-30T06:43:09.280ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
65582025-04-30T06:43:09.280ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
65592025-04-30T06:43:09.282ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e9 19 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62057 req_id = a02e7824-dbb0-4c74-a567-a420272f0ed4
65602025-04-30T06:43:09.282ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59673,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62057 req_id = a02e7824-dbb0-4c74-a567-a420272f0ed4
65612025-04-30T06:43:09.283ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
65622025-04-30T06:43:09.283ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
65632025-04-30T06:43:09.283ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62057 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a02e7824-dbb0-4c74-a567-a420272f0ed4
65642025-04-30T06:43:09.284ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4254, tv_nsec: 83219900 } })
65652025-04-30T06:43:09.284ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
65662025-04-30T06:43:09.284ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: dc 6e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .n..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64340 req_id = a8052fc6-6a58-439a-84ef-3689005fc74b
65672025-04-30T06:43:09.285ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56430,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64340 req_id = a8052fc6-6a58-439a-84ef-3689005fc74b
65682025-04-30T06:43:09.285ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
65692025-04-30T06:43:09.285ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64340 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a8052fc6-6a58-439a-84ef-3689005fc74b
65702025-04-30T06:43:09.357ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
65712025-04-30T06:43:09.357ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65722025-04-30T06:43:09.357ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65732025-04-30T06:43:09.357ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65742025-04-30T06:43:09.357ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65752025-04-30T06:43:09.358ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65762025-04-30T06:43:09.358ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8a 0a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52126 req_id = 27e89d89-f1d9-4d8d-b196-ac9313615cd7
65772025-04-30T06:43:09.359ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35338,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52126 req_id = 27e89d89-f1d9-4d8d-b196-ac9313615cd7
65782025-04-30T06:43:09.359ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
65792025-04-30T06:43:09.360ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35338, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52126 req_id = 27e89d89-f1d9-4d8d-b196-ac9313615cd7
65802025-04-30T06:43:09.360ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
65812025-04-30T06:43:09.360ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
65822025-04-30T06:43:09.360ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
65832025-04-30T06:43:09.361ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65842025-04-30T06:43:09.369ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65852025-04-30T06:43:09.369ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65862025-04-30T06:43:09.369ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65872025-04-30T06:43:09.370ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65882025-04-30T06:43:09.372ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "46311c7c-5a40-4df8-8b2a-9083cf847173", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
65892025-04-30T06:43:09.372ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
65902025-04-30T06:43:09.373ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65912025-04-30T06:43:09.373ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65922025-04-30T06:43:09.373ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65932025-04-30T06:43:09.375ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03f41d10-60a7-406e-bbcd-2a1656adcbc1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
65942025-04-30T06:43:09.375ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
65952025-04-30T06:43:09.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb823fd6-45df-4603-b8f9-f0c92c950800", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
65962025-04-30T06:43:09.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
65972025-04-30T06:43:09.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7ec66cd-ea82-4fef-89f2-30caaa6397a2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
65982025-04-30T06:43:09.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
65992025-04-30T06:43:09.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0270ac57-312a-4aa4-9d5c-d85a88ce68ec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66002025-04-30T06:43:09.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
66012025-04-30T06:43:09.389ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "544fbe07-d14d-4122-96d7-557c7d5f6a66", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66022025-04-30T06:43:09.389ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
66032025-04-30T06:43:09.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb6b56a0-713b-4bd3-8c1d-1bcbb29d81e5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66042025-04-30T06:43:09.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
66052025-04-30T06:43:09.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "972f4578-350e-4e29-91a9-569dd32c3e5c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66062025-04-30T06:43:09.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
66072025-04-30T06:43:09.397ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ddab312-640a-4943-aa0c-2c7762d1c222", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66082025-04-30T06:43:09.397ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
66092025-04-30T06:43:09.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a05dd158-9dc6-44e4-9a3f-e12b254cf892", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66102025-04-30T06:43:09.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
66112025-04-30T06:43:09.404ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
66122025-04-30T06:43:09.407ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc4b481d-c48f-4a64-96bf-2176db7be5c2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66132025-04-30T06:43:09.408ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
66142025-04-30T06:43:09.409ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a0db2b-afac-485f-b128-30aca85a547f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66152025-04-30T06:43:09.409ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
66162025-04-30T06:43:09.410ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aede617-598a-40fd-964a-d8c237918b35", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66172025-04-30T06:43:09.411ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
66182025-04-30T06:43:09.413ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8b1bb14-6515-4bb6-90fe-fcb31bcaa9b9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66192025-04-30T06:43:09.413ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
66202025-04-30T06:43:09.415ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a43cc09-8287-4942-b1ee-26f21a5f6df2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66212025-04-30T06:43:09.415ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
66222025-04-30T06:43:09.417ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3592b30a-2840-4068-9b95-fd1dd65dd7d8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66232025-04-30T06:43:09.419ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
66242025-04-30T06:43:09.423ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b50e1c57-5346-4330-bd28-11458ef95e36", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66252025-04-30T06:43:09.424ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
66262025-04-30T06:43:09.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6840ea1-86d5-4670-8eb4-deb428ef0e71", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66272025-04-30T06:43:09.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
66282025-04-30T06:43:09.430ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09ef8304-c995-4eac-965b-6ed2197d38d5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66292025-04-30T06:43:09.430ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
66302025-04-30T06:43:09.431ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "266b8421-fc93-4218-8b90-5c1607e90155", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66312025-04-30T06:43:09.431ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
66322025-04-30T06:43:09.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1a81d62-0b24-4054-9470-d9ad81452b1b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66332025-04-30T06:43:09.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
66342025-04-30T06:43:09.437ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3caad762-191d-4405-9358-8030c6478f4b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66352025-04-30T06:43:09.437ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
66362025-04-30T06:43:09.439ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c38fa0a-f423-48ec-a9f7-681f9a41ffaa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66372025-04-30T06:43:09.439ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
66382025-04-30T06:43:09.440ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aa71359-7c1a-4957-a9fd-cce316a0abba", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66392025-04-30T06:43:09.440ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
66402025-04-30T06:43:09.443ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "173c2aea-1b8a-456c-acfa-a91d04b317c8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66412025-04-30T06:43:09.444ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
66422025-04-30T06:43:09.449ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8561bd4-6fc9-4178-aa6f-fa45d3052945", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66432025-04-30T06:43:09.449ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
66442025-04-30T06:43:09.453ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90ca6411-3ff6-47dd-8118-c6bcdc392317", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66452025-04-30T06:43:09.453ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
66462025-04-30T06:43:09.456ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66472025-04-30T06:43:09.456ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66482025-04-30T06:43:09.456ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66492025-04-30T06:43:09.456ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
66502025-04-30T06:43:09.457ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66512025-04-30T06:43:09.458ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09303545-d0fc-48af-bf13-62510fec6d01", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66522025-04-30T06:43:09.458ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
66532025-04-30T06:43:09.462ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66542025-04-30T06:43:09.469ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66552025-04-30T06:43:09.469ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66562025-04-30T06:43:09.469ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
66572025-04-30T06:43:09.470ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66582025-04-30T06:43:09.471ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9059376-1598-409a-a84b-9c6693d4bea5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66592025-04-30T06:43:09.471ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
66602025-04-30T06:43:09.472ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66612025-04-30T06:43:09.472ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
66622025-04-30T06:43:09.473ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66632025-04-30T06:43:09.475ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28431587-cfa0-4d89-a1ba-fa41e884a8c4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66642025-04-30T06:43:09.476ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
66652025-04-30T06:43:09.477ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cee94bb-a4fe-46b1-a3d7-2c2fdeabc511", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66662025-04-30T06:43:09.478ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
66672025-04-30T06:43:09.480ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c72a0eb5-0905-45c3-80ca-51af9aa60e2d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66682025-04-30T06:43:09.480ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
66692025-04-30T06:43:09.481ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7e031ff-b337-4f55-9308-684678d517be", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66702025-04-30T06:43:09.481ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
66712025-04-30T06:43:09.482ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a76a6c32-3170-40ea-82de-0547663ef114", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66722025-04-30T06:43:09.482ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
66732025-04-30T06:43:09.484ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3c823f9-8ad3-4ee6-84dd-783190816bc9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66742025-04-30T06:43:09.484ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
66752025-04-30T06:43:09.488ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c76fef7-99bf-4562-a30f-47d6e289d363", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66762025-04-30T06:43:09.488ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
66772025-04-30T06:43:09.491ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92156527-3081-40c0-9c1a-394945dfc6d9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66782025-04-30T06:43:09.491ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
66792025-04-30T06:43:09.495ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44978402-dc79-4ea9-b023-c65773cd47ff", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66802025-04-30T06:43:09.495ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
66812025-04-30T06:43:09.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca098124-45b6-4463-917e-230a3f8dcd6d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66822025-04-30T06:43:09.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
66832025-04-30T06:43:09.550ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91fc6950-7884-402c-a67d-caaada2fd0e8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66842025-04-30T06:43:09.550ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
66852025-04-30T06:43:09.551ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
66862025-04-30T06:43:09.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04d216b1-3a15-4439-9178-79f07875f9a5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66872025-04-30T06:43:09.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
66882025-04-30T06:43:09.559ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17a2d808-5110-442a-a7d8-f039a793f5ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66892025-04-30T06:43:09.559ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
66902025-04-30T06:43:09.560ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e3efb99-571c-4b96-8198-583266518a76", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66912025-04-30T06:43:09.562ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
66922025-04-30T06:43:09.564ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a98645f-f7dd-44d0-bf0c-f1301494cf36", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66932025-04-30T06:43:09.564ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
66942025-04-30T06:43:09.566ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba91dd6b-ae93-4a38-9a0f-7a10cac3c0d0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
66952025-04-30T06:43:09.566ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66962025-04-30T06:43:09.566ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66972025-04-30T06:43:09.567ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66982025-04-30T06:43:09.567ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66992025-04-30T06:43:09.567ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67002025-04-30T06:43:09.570ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67012025-04-30T06:43:09.577ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67022025-04-30T06:43:09.577ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67032025-04-30T06:43:09.577ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67042025-04-30T06:43:09.577ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67052025-04-30T06:43:09.577ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67062025-04-30T06:43:09.577ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67072025-04-30T06:43:09.580ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67082025-04-30T06:43:09.584ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67092025-04-30T06:43:09.584ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67102025-04-30T06:43:09.584ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67112025-04-30T06:43:09.585ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67122025-04-30T06:43:09.586ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
67132025-04-30T06:43:09.586ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67142025-04-30T06:43:09.586ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75586936-0230-4246-a9d0-1d3863efb9b5 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67152025-04-30T06:43:09.587ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75586936-0230-4246-a9d0-1d3863efb9b5 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67162025-04-30T06:43:09.587ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75586936-0230-4246-a9d0-1d3863efb9b5 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67172025-04-30T06:43:09.587ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75586936-0230-4246-a9d0-1d3863efb9b5 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67182025-04-30T06:43:09.651ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
67192025-04-30T06:43:09.651ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
67202025-04-30T06:43:09.652ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 65798 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75586936-0230-4246-a9d0-1d3863efb9b5 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67212025-04-30T06:43:09.653ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67222025-04-30T06:43:09.653ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67232025-04-30T06:43:09.653ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67242025-04-30T06:43:09.653ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
67252025-04-30T06:43:09.654ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67262025-04-30T06:43:09.654ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:42743 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
67272025-04-30T06:43:09.654ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "75586936-0230-4246-a9d0-1d3863efb9b5", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
67282025-04-30T06:43:09.655ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
67292025-04-30T06:43:09.656ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 46ea0493-e42f-493c-865b-621296cdeefd uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
67302025-04-30T06:43:09.656ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:42743 req_id = 5415bf95-2c6e-4e9e-961b-1748619b9b74 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
67312025-04-30T06:43:09.657ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 46ea0493-e42f-493c-865b-621296cdeefd resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
67322025-04-30T06:43:09.657ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 46ea0493-e42f-493c-865b-621296cdeefd roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
67332025-04-30T06:43:09.657ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 46ea0493-e42f-493c-865b-621296cdeefd resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
67342025-04-30T06:43:09.658ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 1584 local_addr = [::1]:51119 method = GET remote_addr = [::1]:42743 req_id = 5415bf95-2c6e-4e9e-961b-1748619b9b74 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
67352025-04-30T06:43:09.659ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5415bf95-2c6e-4e9e-961b-1748619b9b74", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
67362025-04-30T06:43:09.661ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
67372025-04-30T06:43:09.661ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
67382025-04-30T06:43:09.662ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
67392025-04-30T06:43:09.663ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
67402025-04-30T06:43:09.663ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67412025-04-30T06:43:09.663ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67422025-04-30T06:43:09.663ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67432025-04-30T06:43:09.664ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67442025-04-30T06:43:09.664ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67452025-04-30T06:43:09.673ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67462025-04-30T06:43:09.673ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67472025-04-30T06:43:09.673ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
67482025-04-30T06:43:09.674ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67492025-04-30T06:43:09.675ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 18637 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 46ea0493-e42f-493c-865b-621296cdeefd response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
67502025-04-30T06:43:09.676ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "46ea0493-e42f-493c-865b-621296cdeefd", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
67512025-04-30T06:43:09.676ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
67522025-04-30T06:43:09.677ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67532025-04-30T06:43:09.677ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
67542025-04-30T06:43:09.677ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67552025-04-30T06:43:09.684ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
67562025-04-30T06:43:09.684ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67572025-04-30T06:43:09.684ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67582025-04-30T06:43:09.685ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67592025-04-30T06:43:09.685ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67602025-04-30T06:43:09.686ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
67612025-04-30T06:43:09.686ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67622025-04-30T06:43:09.686ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67632025-04-30T06:43:09.686ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67642025-04-30T06:43:09.686ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
67652025-04-30T06:43:09.688ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
67662025-04-30T06:43:09.693ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
67672025-04-30T06:43:09.693ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
67682025-04-30T06:43:09.699ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
67692025-04-30T06:43:09.699ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67702025-04-30T06:43:09.723ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67712025-04-30T06:43:09.723ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67722025-04-30T06:43:09.734ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
67732025-04-30T06:43:09.735ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
67742025-04-30T06:43:09.735ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 482.267116ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
67752025-04-30T06:43:09.735ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
67762025-04-30T06:43:09.735ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
67772025-04-30T06:43:09.735ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67782025-04-30T06:43:09.735ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67792025-04-30T06:43:09.735ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67802025-04-30T06:43:09.736ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67812025-04-30T06:43:09.740ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67822025-04-30T06:43:09.740ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67832025-04-30T06:43:09.740ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67842025-04-30T06:43:09.740ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67852025-04-30T06:43:09.742ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67862025-04-30T06:43:09.750ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
67872025-04-30T06:43:09.750ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67882025-04-30T06:43:09.750ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67892025-04-30T06:43:09.752ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67902025-04-30T06:43:09.757ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
67912025-04-30T06:43:09.758ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
67922025-04-30T06:43:09.759ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
67932025-04-30T06:43:09.760ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 34 72 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 4r..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50779 req_id = 52371e99-f4b7-4ee3-ae45-ed5bf3da0579
67942025-04-30T06:43:09.760ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13426,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50779 req_id = 52371e99-f4b7-4ee3-ae45-ed5bf3da0579
67952025-04-30T06:43:09.761ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
67962025-04-30T06:43:09.762ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
67972025-04-30T06:43:09.762ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50779 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 52371e99-f4b7-4ee3-ae45-ed5bf3da0579
67982025-04-30T06:43:09.762ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4254, tv_nsec: 560938253 } })
67992025-04-30T06:43:09.762ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
68002025-04-30T06:43:09.762ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 19 50 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .P..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49502 req_id = a58c600b-a3da-497d-ad4d-d4e163039e4b
68012025-04-30T06:43:09.762ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6480,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49502 req_id = a58c600b-a3da-497d-ad4d-d4e163039e4b
68022025-04-30T06:43:09.762ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68032025-04-30T06:43:09.762ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49502 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a58c600b-a3da-497d-ad4d-d4e163039e4b
68042025-04-30T06:43:09.857ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
68052025-04-30T06:43:09.858ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68062025-04-30T06:43:09.858ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68072025-04-30T06:43:09.858ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68082025-04-30T06:43:09.858ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
68092025-04-30T06:43:09.859ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68102025-04-30T06:43:09.859ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
68112025-04-30T06:43:09.860ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3d 02 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c =............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53354 req_id = 58bc465b-fba2-44f0-9981-0959b6129eb8
68122025-04-30T06:43:09.860ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15618,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53354 req_id = 58bc465b-fba2-44f0-9981-0959b6129eb8
68132025-04-30T06:43:09.860ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
68142025-04-30T06:43:09.860ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15618, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53354 req_id = 58bc465b-fba2-44f0-9981-0959b6129eb8
68152025-04-30T06:43:09.862ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
68162025-04-30T06:43:09.862ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
68172025-04-30T06:43:09.862ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
68182025-04-30T06:43:09.862ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68192025-04-30T06:43:09.874ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68202025-04-30T06:43:09.874ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68212025-04-30T06:43:09.874ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
68222025-04-30T06:43:09.875ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68232025-04-30T06:43:09.881ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68242025-04-30T06:43:09.881ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
68252025-04-30T06:43:09.882ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68262025-04-30T06:43:09.887ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8488bbe0-597b-4421-883a-7ffb28f6e2c4", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68272025-04-30T06:43:09.887ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
68282025-04-30T06:43:09.896ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b0d38ec-62c0-43a8-8a87-27afac60e9e7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68292025-04-30T06:43:09.897ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
68302025-04-30T06:43:09.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ce2b455-ebf3-4eb7-9a99-3974f1b66c35", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68312025-04-30T06:43:09.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
68322025-04-30T06:43:09.912ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6354933-872a-4bf9-ae1a-e7e845403a16", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68332025-04-30T06:43:09.912ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
68342025-04-30T06:43:09.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd3e93d8-5597-4197-96eb-bf85e8d3327e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68352025-04-30T06:43:09.921ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
68362025-04-30T06:43:09.925ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c9022fd-5050-4dcd-919e-bcdc43b99dbe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68372025-04-30T06:43:09.925ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
68382025-04-30T06:43:09.954ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
68392025-04-30T06:43:09.956ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fe162ff-3744-4524-bb05-e5a39237b4b8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68402025-04-30T06:43:09.956ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
68412025-04-30T06:43:09.966ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23278a60-390b-466b-ab03-8aa875be2bff", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68422025-04-30T06:43:09.966ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
68432025-04-30T06:43:09.968ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c1a28a6-d6bc-45bb-a5b0-fe13bb73f550", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68442025-04-30T06:43:09.968ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
68452025-04-30T06:43:09.969ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67881baa-7999-4e4e-a9e7-fae914368bcb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68462025-04-30T06:43:09.971ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
68472025-04-30T06:43:09.972ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0c64690-b7c6-446e-999b-b5237eb72f90", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68482025-04-30T06:43:09.973ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
68492025-04-30T06:43:09.975ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92a2bfa7-ec9a-4f54-9f86-5d9bcfeb21b7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68502025-04-30T06:43:09.976ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
68512025-04-30T06:43:09.978ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c02f075-6bfd-4aef-b265-e8facc67d193", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68522025-04-30T06:43:09.978ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
68532025-04-30T06:43:09.996ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "475b25a9-1607-4cd3-8c28-eaa2da4c1b97", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68542025-04-30T06:43:09.996ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
68552025-04-30T06:43:10.011ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68562025-04-30T06:43:10.012ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68572025-04-30T06:43:10.012ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68582025-04-30T06:43:10.012ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
68592025-04-30T06:43:10.013ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68602025-04-30T06:43:10.028ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68612025-04-30T06:43:10.034ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68622025-04-30T06:43:10.034ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68632025-04-30T06:43:10.034ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
68642025-04-30T06:43:10.034ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68652025-04-30T06:43:10.039ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68662025-04-30T06:43:10.040ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
68672025-04-30T06:43:10.041ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68682025-04-30T06:43:10.044ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6c19119-9a30-4834-82c8-67a7c1b820c5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68692025-04-30T06:43:10.045ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
68702025-04-30T06:43:10.048ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "118826d0-c14f-41a7-8b64-718cc6e370ae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68712025-04-30T06:43:10.052ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
68722025-04-30T06:43:10.063ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f38629f-1ad4-4095-a7dc-b95452f053d1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68732025-04-30T06:43:10.064ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
68742025-04-30T06:43:10.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86f3edff-bb4b-4fff-9a29-5ccb4cc978b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68752025-04-30T06:43:10.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
68762025-04-30T06:43:10.097ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d31e6df-ee90-4c74-915a-793e05cfdc7f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68772025-04-30T06:43:10.097ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
68782025-04-30T06:43:10.098ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f64e9889-d809-45ab-959a-07342bb9941b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68792025-04-30T06:43:10.098ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
68802025-04-30T06:43:10.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "165a6b55-750f-4cb0-aa85-5dd3d2530ad8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68812025-04-30T06:43:10.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
68822025-04-30T06:43:10.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e7e2755-73e1-4953-afa6-cfe80c4db66b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68832025-04-30T06:43:10.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
68842025-04-30T06:43:10.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "549710ba-ce91-4da5-8933-2a19fa15ec64", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68852025-04-30T06:43:10.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
68862025-04-30T06:43:10.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d987c9c0-d641-445f-9c81-7ddc1dc66a23", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68872025-04-30T06:43:10.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
68882025-04-30T06:43:10.123ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
68892025-04-30T06:43:10.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b908948e-2ef9-40a5-8971-7e7036d97b85", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68902025-04-30T06:43:10.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
68912025-04-30T06:43:10.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88f7e686-4ea8-445b-bddf-c729896d0db6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68922025-04-30T06:43:10.133ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
68932025-04-30T06:43:10.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02b4940f-4200-4fab-8133-f6bfca0ab5ca", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68942025-04-30T06:43:10.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
68952025-04-30T06:43:10.140ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2f69bee-1650-4eeb-be3c-3d125f05d12a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68962025-04-30T06:43:10.140ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
68972025-04-30T06:43:10.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fedf7ad6-99c0-4637-a690-e4946a5a9843", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
68982025-04-30T06:43:10.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
68992025-04-30T06:43:10.145ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f804a87-3ead-42dd-a1f4-cce628ce3372", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69002025-04-30T06:43:10.145ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
69012025-04-30T06:43:10.146ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fbcaeca-fd48-4be2-8c7a-39a1a65b6506", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69022025-04-30T06:43:10.146ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
69032025-04-30T06:43:10.148ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9c571d2-1d9d-4c6c-bebd-78f5a4793711", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69042025-04-30T06:43:10.148ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
69052025-04-30T06:43:10.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a759d1-41ab-41cd-931e-84fabfb5a2f9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69062025-04-30T06:43:10.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
69072025-04-30T06:43:10.150ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee817a78-d3a8-4645-83e3-c8002b29466b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69082025-04-30T06:43:10.151ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
69092025-04-30T06:43:10.161ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62b511e0-1c13-4ddd-ae26-f0ea83775dc1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69102025-04-30T06:43:10.169ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
69112025-04-30T06:43:10.174ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69122025-04-30T06:43:10.174ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69132025-04-30T06:43:10.174ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69142025-04-30T06:43:10.174ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69152025-04-30T06:43:10.175ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69162025-04-30T06:43:10.178ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bd782e2-04ae-4149-856e-46265491d945", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69172025-04-30T06:43:10.178ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
69182025-04-30T06:43:10.182ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69192025-04-30T06:43:10.188ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69202025-04-30T06:43:10.188ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69212025-04-30T06:43:10.188ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69222025-04-30T06:43:10.189ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69232025-04-30T06:43:10.200ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69242025-04-30T06:43:10.201ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69252025-04-30T06:43:10.208ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69262025-04-30T06:43:10.211ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1e1ca8e-2d7b-4395-84ca-a0087e54dfa9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69272025-04-30T06:43:10.211ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
69282025-04-30T06:43:10.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a14af2e0-9cb8-4efb-92dc-aefcbc3cc793", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69292025-04-30T06:43:10.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
69302025-04-30T06:43:10.220ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b674771-d3ad-4e9f-aa3c-1d64ce7df09d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69312025-04-30T06:43:10.220ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
69322025-04-30T06:43:10.222ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a5e750f-edfc-48e6-a0f2-412864b217b1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69332025-04-30T06:43:10.223ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
69342025-04-30T06:43:10.225ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fe150c4-a3d7-48c3-a1f9-8f54191a64b1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69352025-04-30T06:43:10.225ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
69362025-04-30T06:43:10.226ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "104e5fdb-b04d-4eb5-8fd3-b81ee153ee26", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69372025-04-30T06:43:10.226ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
69382025-04-30T06:43:10.227ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ed92709-73ba-4de7-b8f5-521427c4d7c6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69392025-04-30T06:43:10.227ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
69402025-04-30T06:43:10.229ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21076bc6-b5c8-4e88-bcad-933728bba063", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69412025-04-30T06:43:10.229ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
69422025-04-30T06:43:10.232ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0eb0e96-03ec-4f06-806c-0a3d0fd85664", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
69432025-04-30T06:43:10.232ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69442025-04-30T06:43:10.232ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69452025-04-30T06:43:10.232ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69462025-04-30T06:43:10.232ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69472025-04-30T06:43:10.233ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69482025-04-30T06:43:10.248ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69492025-04-30T06:43:10.257ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69502025-04-30T06:43:10.257ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69512025-04-30T06:43:10.257ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69522025-04-30T06:43:10.258ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69532025-04-30T06:43:10.258ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69542025-04-30T06:43:10.258ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69552025-04-30T06:43:10.262ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
69562025-04-30T06:43:10.263ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69572025-04-30T06:43:10.267ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69582025-04-30T06:43:10.267ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69592025-04-30T06:43:10.267ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69602025-04-30T06:43:10.267ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69612025-04-30T06:43:10.381ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
69622025-04-30T06:43:10.381ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
69632025-04-30T06:43:10.385ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69642025-04-30T06:43:10.385ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69652025-04-30T06:43:10.386ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69662025-04-30T06:43:10.386ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69672025-04-30T06:43:10.386ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69682025-04-30T06:43:10.387ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:37669 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
69692025-04-30T06:43:10.400ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:37669 req_id = 8314183e-b29b-4ee6-8351-c2dfeab923a8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
69702025-04-30T06:43:10.402ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 768 local_addr = [::1]:51119 method = GET remote_addr = [::1]:37669 req_id = 8314183e-b29b-4ee6-8351-c2dfeab923a8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
69712025-04-30T06:43:10.402ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8314183e-b29b-4ee6-8351-c2dfeab923a8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
69722025-04-30T06:43:10.404ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
69732025-04-30T06:43:10.404ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
69742025-04-30T06:43:10.404ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
69752025-04-30T06:43:10.416ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
69762025-04-30T06:43:10.417ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69772025-04-30T06:43:10.417ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69782025-04-30T06:43:10.417ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69792025-04-30T06:43:10.418ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69802025-04-30T06:43:10.419ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69812025-04-30T06:43:10.426ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69822025-04-30T06:43:10.426ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69832025-04-30T06:43:10.427ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69842025-04-30T06:43:10.428ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69852025-04-30T06:43:10.452ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69862025-04-30T06:43:10.456ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
69872025-04-30T06:43:10.456ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69882025-04-30T06:43:10.457ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69892025-04-30T06:43:10.457ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69902025-04-30T06:43:10.463ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69912025-04-30T06:43:10.463ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69922025-04-30T06:43:10.463ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69932025-04-30T06:43:10.514ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
69942025-04-30T06:43:10.514ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
69952025-04-30T06:43:10.514ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 779.396537ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
69962025-04-30T06:43:10.514ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
69972025-04-30T06:43:10.514ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
69982025-04-30T06:43:10.515ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69992025-04-30T06:43:10.515ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70002025-04-30T06:43:10.515ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70012025-04-30T06:43:10.515ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70022025-04-30T06:43:10.519ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
70032025-04-30T06:43:10.519ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70042025-04-30T06:43:10.523ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70052025-04-30T06:43:10.523ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70062025-04-30T06:43:10.523ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70072025-04-30T06:43:10.523ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70082025-04-30T06:43:10.527ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
70092025-04-30T06:43:10.527ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
70102025-04-30T06:43:10.528ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 35 83 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 5............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56799 req_id = 7a438c9c-4afb-4f84-afc8-954bca53d494
70112025-04-30T06:43:10.528ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13699,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56799 req_id = 7a438c9c-4afb-4f84-afc8-954bca53d494
70122025-04-30T06:43:10.528ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
70132025-04-30T06:43:10.528ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70142025-04-30T06:43:10.528ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56799 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7a438c9c-4afb-4f84-afc8-954bca53d494
70152025-04-30T06:43:10.529ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4255, tv_nsec: 328015294 } })
70162025-04-30T06:43:10.529ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
70172025-04-30T06:43:10.529ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2a db 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e *...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52078 req_id = 5f943b58-3e45-4a75-bc0d-afd750576d68
70182025-04-30T06:43:10.529ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10971,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52078 req_id = 5f943b58-3e45-4a75-bc0d-afd750576d68
70192025-04-30T06:43:10.529ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70202025-04-30T06:43:10.529ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52078 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5f943b58-3e45-4a75-bc0d-afd750576d68
70212025-04-30T06:43:10.577ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
70222025-04-30T06:43:10.577ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70232025-04-30T06:43:10.577ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70242025-04-30T06:43:10.577ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70252025-04-30T06:43:10.577ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70262025-04-30T06:43:10.578ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70272025-04-30T06:43:10.578ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: aa e0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58004 req_id = 63c326a5-c64a-4ce6-bb73-26d4c651b40b
70282025-04-30T06:43:10.578ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43744,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58004 req_id = 63c326a5-c64a-4ce6-bb73-26d4c651b40b
70292025-04-30T06:43:10.579ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
70302025-04-30T06:43:10.579ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43744, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58004 req_id = 63c326a5-c64a-4ce6-bb73-26d4c651b40b
70312025-04-30T06:43:10.579ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
70322025-04-30T06:43:10.579ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
70332025-04-30T06:43:10.579ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
70342025-04-30T06:43:10.581ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70352025-04-30T06:43:10.587ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70362025-04-30T06:43:10.587ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70372025-04-30T06:43:10.587ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70382025-04-30T06:43:10.588ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70392025-04-30T06:43:10.588ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4212b900-8ebb-44e3-a276-31ee14ce944d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70402025-04-30T06:43:10.588ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
70412025-04-30T06:43:10.590ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "378b2a51-b635-494d-a825-aba961815079", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70422025-04-30T06:43:10.590ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
70432025-04-30T06:43:10.591ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70442025-04-30T06:43:10.591ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70452025-04-30T06:43:10.592ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70462025-04-30T06:43:10.595ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cdd3e45-1b57-4f6b-9a60-111069e30153", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70472025-04-30T06:43:10.595ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
70482025-04-30T06:43:10.597ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc2f8b53-8fde-4bfe-98e1-e668f8fa1dff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70492025-04-30T06:43:10.597ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
70502025-04-30T06:43:10.599ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15fafaac-33f9-4ce4-96e9-cc2c358a641e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70512025-04-30T06:43:10.599ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
70522025-04-30T06:43:10.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b4fe760-3778-4897-b7a2-284ce2d3d6e6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70532025-04-30T06:43:10.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
70542025-04-30T06:43:10.606ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cf28830-a539-4e34-8b8b-8b95dab351f8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70552025-04-30T06:43:10.606ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
70562025-04-30T06:43:10.611ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "caeed0e5-fb8a-4064-bde0-5261bb5dfd13", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70572025-04-30T06:43:10.612ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
70582025-04-30T06:43:10.618ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "acfd58ec-50d7-4a11-9193-534e005988f3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70592025-04-30T06:43:10.618ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
70602025-04-30T06:43:10.620ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72410050-ad73-4130-9b06-e25e60f1aa86", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70612025-04-30T06:43:10.621ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
70622025-04-30T06:43:10.622ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64ac9b6c-826d-4c8f-8750-44e593bce816", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70632025-04-30T06:43:10.623ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
70642025-04-30T06:43:10.625ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "170358d8-a5fc-48b8-8cb4-577d3bd901f6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70652025-04-30T06:43:10.625ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
70662025-04-30T06:43:10.630ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "597259d7-8745-4bc1-9cc0-be0821716f56", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70672025-04-30T06:43:10.630ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
70682025-04-30T06:43:10.631ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
70692025-04-30T06:43:10.643ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "252f6158-b62d-4935-ba0b-2fd07ffc6456", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70702025-04-30T06:43:10.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
70712025-04-30T06:43:10.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "130b4bf7-c0aa-410d-ab50-bc2c9fb13cd2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70722025-04-30T06:43:10.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
70732025-04-30T06:43:10.652ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9f65a29-1664-401a-bc9b-f7973bab00e2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70742025-04-30T06:43:10.653ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
70752025-04-30T06:43:10.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5859a4-be59-460a-b7bc-e41a8c4cdac0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70762025-04-30T06:43:10.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
70772025-04-30T06:43:10.658ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8258bf91-9d42-4caf-ba03-8d43bddee241", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70782025-04-30T06:43:10.658ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
70792025-04-30T06:43:10.659ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08664936-2a4b-4631-9a04-278ccd54b411", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70802025-04-30T06:43:10.659ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
70812025-04-30T06:43:10.660ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55977f18-8629-478b-8b9e-110dd3c42148", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70822025-04-30T06:43:10.660ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
70832025-04-30T06:43:10.696ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70842025-04-30T06:43:10.696ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70852025-04-30T06:43:10.696ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70862025-04-30T06:43:10.696ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70872025-04-30T06:43:10.697ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70882025-04-30T06:43:10.697ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
70892025-04-30T06:43:10.697ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
70902025-04-30T06:43:10.698ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
70912025-04-30T06:43:10.700ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
70922025-04-30T06:43:10.701ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31776fae-b626-4c0b-984b-b50e82285377", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
70932025-04-30T06:43:10.701ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
70942025-04-30T06:43:10.702ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70952025-04-30T06:43:10.712ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70962025-04-30T06:43:10.713ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70972025-04-30T06:43:10.713ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70982025-04-30T06:43:10.713ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70992025-04-30T06:43:10.739ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71002025-04-30T06:43:10.739ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71012025-04-30T06:43:10.741ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
71022025-04-30T06:43:10.741ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
71032025-04-30T06:43:10.742ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0414ffa-3faf-4f07-8baa-5a11f61093dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71042025-04-30T06:43:10.742ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
71052025-04-30T06:43:10.743ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71062025-04-30T06:43:10.743ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
71072025-04-30T06:43:10.744ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71082025-04-30T06:43:10.744ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71092025-04-30T06:43:10.745ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71102025-04-30T06:43:10.745ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71112025-04-30T06:43:10.745ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71122025-04-30T06:43:10.749ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
71132025-04-30T06:43:10.749ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71142025-04-30T06:43:10.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93a6201d-7407-488f-a11a-6bf12ee546a6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71152025-04-30T06:43:10.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
71162025-04-30T06:43:10.753ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eb0c6c2-32cd-45ce-a3c8-312c98619e92", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71172025-04-30T06:43:10.753ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
71182025-04-30T06:43:10.772ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "407b437f-f78a-45cb-94dd-bd3348ea6b61", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71192025-04-30T06:43:10.772ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
71202025-04-30T06:43:10.777ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd1f58e7-b19f-4359-8439-c61ad9142176", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71212025-04-30T06:43:10.777ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
71222025-04-30T06:43:10.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4450044f-1735-44a9-a21b-673d93e723a3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71232025-04-30T06:43:10.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
71242025-04-30T06:43:10.783ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a255f0f9-09dd-49f7-89e9-7e8a686de655", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71252025-04-30T06:43:10.783ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
71262025-04-30T06:43:10.787ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7929fcb0-d647-4498-b42a-eb5083cf57bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71272025-04-30T06:43:10.787ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
71282025-04-30T06:43:10.788ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
71292025-04-30T06:43:10.791ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77362a6c-dc4b-43f7-a111-36ddf5bf995c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71302025-04-30T06:43:10.791ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
71312025-04-30T06:43:10.792ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91c1c372-d451-4042-91e8-4ec9238f341e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71322025-04-30T06:43:10.793ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
71332025-04-30T06:43:10.793ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "674f1fce-9d57-40ab-a827-5a4b5b26c631", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71342025-04-30T06:43:10.793ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
71352025-04-30T06:43:10.794ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e58a692e-ec5b-4766-b156-6de299e8af8b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71362025-04-30T06:43:10.794ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
71372025-04-30T06:43:10.796ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "beb91484-08f4-42ff-8ee7-060869547f76", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71382025-04-30T06:43:10.796ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
71392025-04-30T06:43:10.801ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
71402025-04-30T06:43:10.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1a9c75f-d745-4d9e-a8f4-51296f85a8a9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71412025-04-30T06:43:10.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
71422025-04-30T06:43:10.805ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f050879d-a316-4943-a677-236f923b6ea4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71432025-04-30T06:43:10.805ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
71442025-04-30T06:43:10.808ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c33b677c-8243-4e6a-b14a-ea396e2429f2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71452025-04-30T06:43:10.810ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
71462025-04-30T06:43:10.811ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f83eb1b2-c724-4e6f-b0a4-61dde84dd967", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71472025-04-30T06:43:10.811ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
71482025-04-30T06:43:10.814ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4bef3af-6cdf-41fa-bcb2-cc77286d0cf9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71492025-04-30T06:43:10.814ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
71502025-04-30T06:43:10.822ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45fd63a8-191e-4967-837b-5a19f03bcfb5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71512025-04-30T06:43:10.827ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
71522025-04-30T06:43:10.831ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85ea26d2-6f48-4eee-a9c3-5fa117f5c68e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71532025-04-30T06:43:10.832ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
71542025-04-30T06:43:10.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3c3b96-5270-4e7d-92ff-dd1d485236ed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71552025-04-30T06:43:10.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
71562025-04-30T06:43:10.834ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c312629-2327-431e-9a54-d933755fd13e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71572025-04-30T06:43:10.834ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
71582025-04-30T06:43:10.835ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bc83be8-f087-444d-af9c-1544f4cd2ea6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71592025-04-30T06:43:10.836ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
71602025-04-30T06:43:10.837ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd45ea65-7de5-4d76-913b-99083e4063c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71612025-04-30T06:43:10.837ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
71622025-04-30T06:43:10.837ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71632025-04-30T06:43:10.837ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71642025-04-30T06:43:10.838ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71652025-04-30T06:43:10.840ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71662025-04-30T06:43:10.840ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71672025-04-30T06:43:10.840ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71682025-04-30T06:43:10.840ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71692025-04-30T06:43:10.840ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
71702025-04-30T06:43:10.841ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71712025-04-30T06:43:10.845ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71722025-04-30T06:43:10.851ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
71732025-04-30T06:43:10.851ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
71742025-04-30T06:43:10.851ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71752025-04-30T06:43:10.851ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71762025-04-30T06:43:10.851ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71772025-04-30T06:43:10.852ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71782025-04-30T06:43:10.858ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71792025-04-30T06:43:10.865ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71802025-04-30T06:43:10.883ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71812025-04-30T06:43:10.883ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
71822025-04-30T06:43:10.884ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71832025-04-30T06:43:10.900ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71842025-04-30T06:43:10.909ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
71852025-04-30T06:43:10.909ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71862025-04-30T06:43:10.909ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71872025-04-30T06:43:10.910ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71882025-04-30T06:43:10.912ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71892025-04-30T06:43:10.912ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
71902025-04-30T06:43:10.913ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71912025-04-30T06:43:11.027ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
71922025-04-30T06:43:11.028ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
71932025-04-30T06:43:11.035ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:33492 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
71942025-04-30T06:43:11.039ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:33492 req_id = 6f2971f9-0f5f-4498-ae14-98b0575d69a0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
71952025-04-30T06:43:11.041ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 1530 local_addr = [::1]:51119 method = GET remote_addr = [::1]:33492 req_id = 6f2971f9-0f5f-4498-ae14-98b0575d69a0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
71962025-04-30T06:43:11.046ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f2971f9-0f5f-4498-ae14-98b0575d69a0", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
71972025-04-30T06:43:11.048ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
71982025-04-30T06:43:11.049ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
71992025-04-30T06:43:11.049ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
72002025-04-30T06:43:11.051ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72012025-04-30T06:43:11.051ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72022025-04-30T06:43:11.052ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72032025-04-30T06:43:11.052ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72042025-04-30T06:43:11.053ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72052025-04-30T06:43:11.067ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72062025-04-30T06:43:11.072ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72072025-04-30T06:43:11.072ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72082025-04-30T06:43:11.073ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72092025-04-30T06:43:11.073ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72102025-04-30T06:43:11.114ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
72112025-04-30T06:43:11.200ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72122025-04-30T06:43:11.200ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72132025-04-30T06:43:11.200ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72142025-04-30T06:43:11.200ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
72152025-04-30T06:43:11.203ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72162025-04-30T06:43:11.205ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
72172025-04-30T06:43:11.205ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
72182025-04-30T06:43:11.205ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 690.735304ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
72192025-04-30T06:43:11.205ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
72202025-04-30T06:43:11.205ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72212025-04-30T06:43:11.205ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72222025-04-30T06:43:11.205ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72232025-04-30T06:43:11.205ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72242025-04-30T06:43:11.206ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72252025-04-30T06:43:11.210ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72262025-04-30T06:43:11.217ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72272025-04-30T06:43:11.217ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72282025-04-30T06:43:11.217ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
72292025-04-30T06:43:11.217ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72302025-04-30T06:43:11.219ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72312025-04-30T06:43:11.224ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72322025-04-30T06:43:11.224ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72332025-04-30T06:43:11.224ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72342025-04-30T06:43:11.225ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72352025-04-30T06:43:11.231ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72362025-04-30T06:43:11.231ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
72372025-04-30T06:43:11.234ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72382025-04-30T06:43:11.241ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
72392025-04-30T06:43:11.241ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
72402025-04-30T06:43:11.242ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8b 8e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57884 req_id = b74c994b-73eb-45c1-a7b8-fa19742a2e31
72412025-04-30T06:43:11.242ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35726,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57884 req_id = b74c994b-73eb-45c1-a7b8-fa19742a2e31
72422025-04-30T06:43:11.242ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
72432025-04-30T06:43:11.242ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72442025-04-30T06:43:11.242ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57884 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b74c994b-73eb-45c1-a7b8-fa19742a2e31
72452025-04-30T06:43:11.242ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4256, tv_nsec: 41737969 } })
72462025-04-30T06:43:11.242ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
72472025-04-30T06:43:11.243ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bb ff 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54411 req_id = 2800817b-802a-492f-b187-008de08962f3
72482025-04-30T06:43:11.243ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48127,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54411 req_id = 2800817b-802a-492f-b187-008de08962f3
72492025-04-30T06:43:11.243ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72502025-04-30T06:43:11.243ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54411 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2800817b-802a-492f-b187-008de08962f3
72512025-04-30T06:43:11.323ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
72522025-04-30T06:43:11.325ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8d 33 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .3..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65094 req_id = 6b64f20d-95fb-4c51-a2e2-62b2f00c82ef
72532025-04-30T06:43:11.325ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36147,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65094 req_id = 6b64f20d-95fb-4c51-a2e2-62b2f00c82ef
72542025-04-30T06:43:11.325ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
72552025-04-30T06:43:11.325ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36147, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65094 req_id = 6b64f20d-95fb-4c51-a2e2-62b2f00c82ef
72562025-04-30T06:43:11.326ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
72572025-04-30T06:43:11.326ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
72582025-04-30T06:43:11.326ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
72592025-04-30T06:43:11.328ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdbffe16-53f2-4adf-b7bd-0adeaf92ec6b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72602025-04-30T06:43:11.328ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
72612025-04-30T06:43:11.329ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad1cea8a-3097-4218-beab-69c5ccf245c4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72622025-04-30T06:43:11.330ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
72632025-04-30T06:43:11.332ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ea94261-5150-4e3c-9b93-00bee8abf178", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72642025-04-30T06:43:11.332ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
72652025-04-30T06:43:11.347ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18c89632-a98c-4136-853d-f4510a1f7620", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72662025-04-30T06:43:11.347ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
72672025-04-30T06:43:11.350ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bc93da3-492f-440f-96c3-36a4d13f4bcd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72682025-04-30T06:43:11.350ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
72692025-04-30T06:43:11.352ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d2697b2-92b1-408c-a293-93a252562ab2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72702025-04-30T06:43:11.352ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
72712025-04-30T06:43:11.361ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b26333de-6c14-4e86-bda1-ae91dfca4997", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72722025-04-30T06:43:11.361ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
72732025-04-30T06:43:11.361ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
72742025-04-30T06:43:11.366ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3617e244-6ae0-4ec3-a8c3-6d188636470c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72752025-04-30T06:43:11.367ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
72762025-04-30T06:43:11.368ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f3f2a16-f7f0-48b0-be78-d51dbec5d01a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72772025-04-30T06:43:11.368ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
72782025-04-30T06:43:11.371ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86415a02-e4c7-4a34-98f2-51d0d60aad1c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72792025-04-30T06:43:11.371ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
72802025-04-30T06:43:11.372ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dcf0357-0065-43cb-babe-a9859ff041f5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72812025-04-30T06:43:11.373ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
72822025-04-30T06:43:11.374ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49707b55-ef45-4ae4-8725-78bd8777ca4d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72832025-04-30T06:43:11.374ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
72842025-04-30T06:43:11.375ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cb96621-4af8-4f9a-acd5-d10620715329", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72852025-04-30T06:43:11.375ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
72862025-04-30T06:43:11.380ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "128e8c06-6880-407a-90bb-86283499766b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72872025-04-30T06:43:11.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
72882025-04-30T06:43:11.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5fc2f98-6afa-401a-b077-6e781ab8f104", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72892025-04-30T06:43:11.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
72902025-04-30T06:43:11.386ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5709e9fe-2a5c-402d-9128-36ef532869e3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72912025-04-30T06:43:11.387ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
72922025-04-30T06:43:11.389ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27a6a414-6639-49c0-8952-425c787ebab4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72932025-04-30T06:43:11.389ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
72942025-04-30T06:43:11.391ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1f643d0-21d3-483e-b00c-920366af72a8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72952025-04-30T06:43:11.391ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
72962025-04-30T06:43:11.393ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed6d5479-cb92-4eac-8709-0d9a705cba03", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72972025-04-30T06:43:11.393ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
72982025-04-30T06:43:11.394ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a14b5c-16ce-4621-99de-12019fb58a19", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
72992025-04-30T06:43:11.394ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
73002025-04-30T06:43:11.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd5ea3e7-dc11-4b0b-9d7f-0e5fa0008e87", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
73012025-04-30T06:43:11.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
73022025-04-30T06:43:11.397ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bc120ce-0785-49c4-82f7-1301af6223e1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
73032025-04-30T06:43:11.397ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
73042025-04-30T06:43:11.398ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d570d266-28af-40ec-9843-e5f131652037", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
73052025-04-30T06:43:11.398ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
73062025-04-30T06:43:11.399ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16ea045a-c331-436b-8699-dd4cf2cf7779", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
73072025-04-30T06:43:11.400ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
73082025-04-30T06:43:11.403ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9351adef-fbd2-4fca-8ad3-5649b53345d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73092025-04-30T06:43:11.403ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
73102025-04-30T06:43:11.403ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e945a710-c3e0-48c0-a985-95e0c7625239", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73112025-04-30T06:43:11.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
73122025-04-30T06:43:11.406ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cc36235-d0b7-4454-8da3-4491d2b0d9f2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73132025-04-30T06:43:11.406ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
73142025-04-30T06:43:11.408ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5adea610-8193-4d10-af08-523a3f760158", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73152025-04-30T06:43:11.408ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
73162025-04-30T06:43:11.411ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df26bcca-339f-4008-83e4-225174272bed", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73172025-04-30T06:43:11.411ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
73182025-04-30T06:43:11.412ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73192025-04-30T06:43:11.412ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73202025-04-30T06:43:11.412ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73212025-04-30T06:43:11.412ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
73222025-04-30T06:43:11.413ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73232025-04-30T06:43:11.417ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73242025-04-30T06:43:11.427ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73252025-04-30T06:43:11.427ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73262025-04-30T06:43:11.427ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
73272025-04-30T06:43:11.427ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73282025-04-30T06:43:11.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38e5884e-583f-4171-bdb4-65862f9a4745", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73292025-04-30T06:43:11.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
73302025-04-30T06:43:11.431ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73312025-04-30T06:43:11.431ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
73322025-04-30T06:43:11.432ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73332025-04-30T06:43:11.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "248bf70b-7b54-4603-945c-cc6a2bc895b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73342025-04-30T06:43:11.434ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
73352025-04-30T06:43:11.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf2a59a4-5a03-4c96-b635-5fcf62111ac6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73362025-04-30T06:43:11.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
73372025-04-30T06:43:11.441ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c35af8a-ed3a-484c-9343-a95b0957b601", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73382025-04-30T06:43:11.441ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
73392025-04-30T06:43:11.443ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "220cf163-4be3-40f8-a839-af5f28dfa2cf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73402025-04-30T06:43:11.443ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
73412025-04-30T06:43:11.454ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec8d251c-5ddf-4f0d-bc5a-c1d0f0b4fb69", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73422025-04-30T06:43:11.455ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
73432025-04-30T06:43:11.459ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea9be17c-6ae4-406c-9893-6f55441e1ecb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73442025-04-30T06:43:11.459ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
73452025-04-30T06:43:11.462ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10d8f5a3-8b5c-4594-a593-2cbf87ae618b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73462025-04-30T06:43:11.462ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
73472025-04-30T06:43:11.466ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0451111a-154f-40bb-8122-f7fb9c0aa4dd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73482025-04-30T06:43:11.466ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
73492025-04-30T06:43:11.493ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
73502025-04-30T06:43:11.493ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfe3f599-31df-41c3-8098-3c65bc1a5a64", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73512025-04-30T06:43:11.494ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
73522025-04-30T06:43:11.498ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50947830-647a-4d45-8eeb-54d8e7d2a892", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73532025-04-30T06:43:11.498ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
73542025-04-30T06:43:11.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac914283-94b8-4ac3-9241-a7e590cdfa5c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73552025-04-30T06:43:11.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
73562025-04-30T06:43:11.502ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38103f02-22c4-4ad4-b8c7-65e6840c5cd7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73572025-04-30T06:43:11.502ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
73582025-04-30T06:43:11.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f77aa7c9-5cb6-4e1f-bebb-a4f65af9c0bc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73592025-04-30T06:43:11.507ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
73602025-04-30T06:43:11.511ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fd1139c-e834-4a11-b47f-82ff4201d410", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73612025-04-30T06:43:11.511ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
73622025-04-30T06:43:11.512ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11112a49-3a84-4e71-a9ef-969fab6a78ab", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
73632025-04-30T06:43:11.512ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
73642025-04-30T06:43:11.513ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73652025-04-30T06:43:11.513ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73662025-04-30T06:43:11.513ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73672025-04-30T06:43:11.513ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73682025-04-30T06:43:11.517ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73692025-04-30T06:43:11.527ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
73702025-04-30T06:43:11.527ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
73712025-04-30T06:43:11.527ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73722025-04-30T06:43:11.527ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73732025-04-30T06:43:11.527ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73742025-04-30T06:43:11.528ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73752025-04-30T06:43:11.530ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73762025-04-30T06:43:11.537ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
73772025-04-30T06:43:11.537ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73782025-04-30T06:43:11.537ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73792025-04-30T06:43:11.538ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73802025-04-30T06:43:11.609ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
73812025-04-30T06:43:11.609ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
73822025-04-30T06:43:11.610ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73832025-04-30T06:43:11.610ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73842025-04-30T06:43:11.610ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73852025-04-30T06:43:11.610ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
73862025-04-30T06:43:11.612ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73872025-04-30T06:43:11.612ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:46623 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
73882025-04-30T06:43:11.612ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
73892025-04-30T06:43:11.614ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73902025-04-30T06:43:11.614ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 20a1a1d7-3015-4b1a-b501-309e64565ae0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73912025-04-30T06:43:11.615ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:46623 req_id = 5bad68a0-6c78-4b9d-bc52-0512ba1dee21 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73922025-04-30T06:43:11.615ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 20a1a1d7-3015-4b1a-b501-309e64565ae0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73932025-04-30T06:43:11.615ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 20a1a1d7-3015-4b1a-b501-309e64565ae0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73942025-04-30T06:43:11.616ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 20a1a1d7-3015-4b1a-b501-309e64565ae0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73952025-04-30T06:43:11.616ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 1735 local_addr = [::1]:51119 method = GET remote_addr = [::1]:46623 req_id = 5bad68a0-6c78-4b9d-bc52-0512ba1dee21 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73962025-04-30T06:43:11.617ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73972025-04-30T06:43:11.628ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73982025-04-30T06:43:11.628ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73992025-04-30T06:43:11.628ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
74002025-04-30T06:43:11.630ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74012025-04-30T06:43:11.630ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bad68a0-6c78-4b9d-bc52-0512ba1dee21", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
74022025-04-30T06:43:11.638ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74032025-04-30T06:43:11.638ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74042025-04-30T06:43:11.638ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
74052025-04-30T06:43:11.645ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74062025-04-30T06:43:11.645ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74072025-04-30T06:43:11.646ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74082025-04-30T06:43:11.646ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74092025-04-30T06:43:11.646ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74102025-04-30T06:43:11.649ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 35028 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 20a1a1d7-3015-4b1a-b501-309e64565ae0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
74112025-04-30T06:43:11.651ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a1a1d7-3015-4b1a-b501-309e64565ae0", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
74122025-04-30T06:43:11.651ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74132025-04-30T06:43:11.651ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74142025-04-30T06:43:11.651ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
74152025-04-30T06:43:11.652ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74162025-04-30T06:43:11.652ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = da455dc3-508a-4d61-b18b-a936bfb8d2ff uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74172025-04-30T06:43:11.653ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = da455dc3-508a-4d61-b18b-a936bfb8d2ff resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74182025-04-30T06:43:11.653ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = da455dc3-508a-4d61-b18b-a936bfb8d2ff roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74192025-04-30T06:43:11.655ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = da455dc3-508a-4d61-b18b-a936bfb8d2ff resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74202025-04-30T06:43:11.656ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74212025-04-30T06:43:11.664ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74222025-04-30T06:43:11.664ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74232025-04-30T06:43:11.664ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74242025-04-30T06:43:11.665ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74252025-04-30T06:43:11.666ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
74262025-04-30T06:43:11.666ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
74272025-04-30T06:43:11.674ZINFOtest_snapshot_expunge (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:47175 remote_addr = 127.0.0.1:55312
74282025-04-30T06:43:11.675ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = 127.0.0.1:47175 method = GET remote_addr = 127.0.0.1:55312 req_id = 3141f585-3fec-4f42-81dd-7ca72882d681 uri = /58f2365f-b720-4ec1-ad50-a68fadcacb1c
74292025-04-30T06:43:11.682ZINFOtest_snapshot_expunge (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 7273 local_addr = 127.0.0.1:47175 method = GET remote_addr = 127.0.0.1:55312 req_id = 3141f585-3fec-4f42-81dd-7ca72882d681 response_code = 200 uri = /58f2365f-b720-4ec1-ad50-a68fadcacb1c
74302025-04-30T06:43:11.683ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 30144 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = da455dc3-508a-4d61-b18b-a936bfb8d2ff response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
74312025-04-30T06:43:11.685ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74322025-04-30T06:43:11.685ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74332025-04-30T06:43:11.685ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "da455dc3-508a-4d61-b18b-a936bfb8d2ff", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
74342025-04-30T06:43:11.685ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
74352025-04-30T06:43:11.685ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74362025-04-30T06:43:11.685ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
74372025-04-30T06:43:11.685ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
74382025-04-30T06:43:11.686ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74392025-04-30T06:43:11.687ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
74402025-04-30T06:43:11.738ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
74412025-04-30T06:43:11.738ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
74422025-04-30T06:43:11.738ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
74432025-04-30T06:43:11.742ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74442025-04-30T06:43:11.742ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74452025-04-30T06:43:11.743ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
74462025-04-30T06:43:11.743ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
74472025-04-30T06:43:11.748ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74482025-04-30T06:43:11.748ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74492025-04-30T06:43:11.749ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74502025-04-30T06:43:11.749ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74512025-04-30T06:43:11.751ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
74522025-04-30T06:43:11.751ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74532025-04-30T06:43:11.751ZDEBGtest_snapshot_expunge (results-sink): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74542025-04-30T06:43:11.752ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 17 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74552025-04-30T06:43:11.753ZDEBGtest_snapshot_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74562025-04-30T06:43:11.753ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
74572025-04-30T06:43:11.757ZTRACtest_snapshot_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74582025-04-30T06:43:11.758ZDEBGtest_snapshot_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:11.753708469Z } timeseries_name = http_service:request_latency_histogram
74592025-04-30T06:43:11.764ZDEBGtest_snapshot_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74602025-04-30T06:43:11.764ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
74612025-04-30T06:43:11.771ZTRACtest_snapshot_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74622025-04-30T06:43:11.771ZDEBGtest_snapshot_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:11.764199101Z } timeseries_name = collection_target:cpus_provisioned
74632025-04-30T06:43:11.772ZDEBGtest_snapshot_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74642025-04-30T06:43:11.773ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
74652025-04-30T06:43:11.776ZTRACtest_snapshot_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74662025-04-30T06:43:11.777ZDEBGtest_snapshot_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:11.772740499Z } timeseries_name = collection_target:ram_provisioned
74672025-04-30T06:43:11.778ZDEBGtest_snapshot_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74682025-04-30T06:43:11.778ZTRACtest_snapshot_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
74692025-04-30T06:43:11.781ZTRACtest_snapshot_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74702025-04-30T06:43:11.781ZDEBGtest_snapshot_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:11.777969057Z } timeseries_name = collection_target:virtual_disk_space_provisioned
74712025-04-30T06:43:11.789ZDEBGtest_snapshot_expunge (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
74722025-04-30T06:43:11.789ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
74732025-04-30T06:43:11.803ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
74742025-04-30T06:43:11.804ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74752025-04-30T06:43:11.804ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
74762025-04-30T06:43:11.805ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_string FORMAT Native
74772025-04-30T06:43:11.806ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
74782025-04-30T06:43:11.807ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
74792025-04-30T06:43:11.807ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 601.604437ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
74802025-04-30T06:43:11.807ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
74812025-04-30T06:43:11.807ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74822025-04-30T06:43:11.808ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74832025-04-30T06:43:11.808ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74842025-04-30T06:43:11.808ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74852025-04-30T06:43:11.808ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74862025-04-30T06:43:11.812ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 10 table_name = fields_string
74872025-04-30T06:43:11.812ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
74882025-04-30T06:43:11.816ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74892025-04-30T06:43:11.822ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74902025-04-30T06:43:11.822ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74912025-04-30T06:43:11.822ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74922025-04-30T06:43:11.822ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74932025-04-30T06:43:11.823ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 5 table_name = fields_u16
74942025-04-30T06:43:11.823ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
74952025-04-30T06:43:11.826ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
74962025-04-30T06:43:11.826ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
74972025-04-30T06:43:11.828ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 17 table_name = fields_uuid
74982025-04-30T06:43:11.828ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
74992025-04-30T06:43:11.828ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: eb 81 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63260 req_id = 2943fc0d-266d-40a6-8479-f8feb49ef66e
75002025-04-30T06:43:11.828ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60289,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63260 req_id = 2943fc0d-266d-40a6-8479-f8feb49ef66e
75012025-04-30T06:43:11.828ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
75022025-04-30T06:43:11.828ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75032025-04-30T06:43:11.828ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63260 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2943fc0d-266d-40a6-8479-f8feb49ef66e
75042025-04-30T06:43:11.829ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4256, tv_nsec: 628138844 } })
75052025-04-30T06:43:11.829ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
75062025-04-30T06:43:11.830ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 77 ac 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e w...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65482 req_id = 2e584bca-b1ef-4f26-8cb4-6c886b9d0340
75072025-04-30T06:43:11.830ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30636,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65482 req_id = 2e584bca-b1ef-4f26-8cb4-6c886b9d0340
75082025-04-30T06:43:11.830ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75092025-04-30T06:43:11.830ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65482 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2e584bca-b1ef-4f26-8cb4-6c886b9d0340
75102025-04-30T06:43:11.878ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
75112025-04-30T06:43:11.879ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75122025-04-30T06:43:11.879ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75132025-04-30T06:43:11.879ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75142025-04-30T06:43:11.879ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75152025-04-30T06:43:11.880ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75162025-04-30T06:43:11.881ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 5 table_name = measurements_histogramu64
75172025-04-30T06:43:11.881ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
75182025-04-30T06:43:11.881ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0e d5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63500 req_id = 7153b9c0-7937-47b0-8a5f-03accf8932b9
75192025-04-30T06:43:11.881ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3797,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63500 req_id = 7153b9c0-7937-47b0-8a5f-03accf8932b9
75202025-04-30T06:43:11.882ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
75212025-04-30T06:43:11.882ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3797, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63500 req_id = 7153b9c0-7937-47b0-8a5f-03accf8932b9
75222025-04-30T06:43:11.882ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
75232025-04-30T06:43:11.882ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
75242025-04-30T06:43:11.883ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
75252025-04-30T06:43:11.885ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 12 table_name = measurements_i64
75262025-04-30T06:43:11.885ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75272025-04-30T06:43:11.885ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75282025-04-30T06:43:11.894ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75292025-04-30T06:43:11.894ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75302025-04-30T06:43:11.894ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75312025-04-30T06:43:11.895ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75322025-04-30T06:43:11.896ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d8eb206-fa74-43dc-81b2-caf291613d45", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75332025-04-30T06:43:11.896ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
75342025-04-30T06:43:11.902ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75352025-04-30T06:43:11.902ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75362025-04-30T06:43:11.904ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75372025-04-30T06:43:11.906ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e87b9dd-fcd9-49fe-9c0d-337af8359ae7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75382025-04-30T06:43:11.906ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
75392025-04-30T06:43:11.906ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
75402025-04-30T06:43:11.909ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06f7879f-ff29-43d0-b758-fe66d34c40ea", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75412025-04-30T06:43:11.909ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
75422025-04-30T06:43:11.913ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48a9d27-be1f-4c5f-b714-df972855bda4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75432025-04-30T06:43:11.913ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
75442025-04-30T06:43:11.916ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8da9434-fea3-4828-bca2-981221471ef8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75452025-04-30T06:43:11.916ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
75462025-04-30T06:43:11.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd2de72f-5488-4524-846c-c9dbd63bea98", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75472025-04-30T06:43:11.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
75482025-04-30T06:43:11.926ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9292aaa5-b257-4ad0-8680-00c122f2e7e9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75492025-04-30T06:43:11.926ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
75502025-04-30T06:43:11.931ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4585bba7-cb71-409f-b459-c5a8f81673be", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75512025-04-30T06:43:11.932ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
75522025-04-30T06:43:11.939ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
75532025-04-30T06:43:11.940ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6cc2702-2752-4410-9afd-f01465e9446b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75542025-04-30T06:43:11.941ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
75552025-04-30T06:43:11.942ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b651d4f-993e-4b71-97d0-a86a32920c8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75562025-04-30T06:43:11.942ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
75572025-04-30T06:43:11.944ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad19b3e-e317-4f08-8f6f-0895b64040bd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75582025-04-30T06:43:11.944ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
75592025-04-30T06:43:11.945ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dc451a2-d735-4456-94f2-b461cf04cd9b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75602025-04-30T06:43:11.945ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
75612025-04-30T06:43:11.947ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be31b603-9c1d-432f-aaca-4bcc91e68aed", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75622025-04-30T06:43:11.947ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
75632025-04-30T06:43:11.950ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d17d49a-5a40-41cb-bb58-a5c9017e0424", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75642025-04-30T06:43:11.951ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
75652025-04-30T06:43:11.956ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc2500e-9bde-4cdb-917c-b001a7eea3f2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75662025-04-30T06:43:11.956ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
75672025-04-30T06:43:11.958ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76c57139-6b83-4bc4-b03e-a927767d2d68", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75682025-04-30T06:43:11.958ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
75692025-04-30T06:43:11.963ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13693f7b-3192-4e42-8eb3-496c81795067", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75702025-04-30T06:43:11.963ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
75712025-04-30T06:43:11.965ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97ff0f5e-e697-4433-a205-54c7e40d2368", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75722025-04-30T06:43:11.965ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
75732025-04-30T06:43:11.968ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e26dd335-b968-4a90-8a8a-bf3f520b286b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75742025-04-30T06:43:11.968ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
75752025-04-30T06:43:11.971ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaf2b5b4-f406-4153-a344-e196fd205557", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75762025-04-30T06:43:11.971ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
75772025-04-30T06:43:11.972ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddd0579c-802e-4303-afcc-bcbbeb5c774d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75782025-04-30T06:43:11.972ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
75792025-04-30T06:43:11.973ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43cce2ec-38ba-4ac7-b6f0-0fe70d28142a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75802025-04-30T06:43:11.973ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
75812025-04-30T06:43:11.982ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46a1033f-75ce-456e-a2dd-dc234edd04cc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75822025-04-30T06:43:11.982ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
75832025-04-30T06:43:11.992ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75842025-04-30T06:43:11.992ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75852025-04-30T06:43:11.992ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75862025-04-30T06:43:11.992ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75872025-04-30T06:43:11.993ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75882025-04-30T06:43:12.003ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75892025-04-30T06:43:12.009ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75902025-04-30T06:43:12.009ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75912025-04-30T06:43:12.009ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75922025-04-30T06:43:12.010ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75932025-04-30T06:43:12.030ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fd2ee36-d86f-4170-979e-07a8357c9b6e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75942025-04-30T06:43:12.030ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
75952025-04-30T06:43:12.035ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75962025-04-30T06:43:12.035ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75972025-04-30T06:43:12.036ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75982025-04-30T06:43:12.094ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83cf9d7a-164a-41ed-8ea1-121ea40815db", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
75992025-04-30T06:43:12.095ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
76002025-04-30T06:43:12.103ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3679efb-994d-45fa-8bb1-9d97230459f4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76012025-04-30T06:43:12.103ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
76022025-04-30T06:43:12.107ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3baf079e-3233-4998-88f1-b7f74f8d37ba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76032025-04-30T06:43:12.108ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
76042025-04-30T06:43:12.113ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f70a2b1c-71c6-4201-ad0e-c4b08bb014f8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76052025-04-30T06:43:12.114ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
76062025-04-30T06:43:12.115ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
76072025-04-30T06:43:12.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b3920c8-dc32-4b90-a664-32a12ad8f690", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76082025-04-30T06:43:12.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
76092025-04-30T06:43:12.125ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e386d032-619c-4f25-8548-7400e235bc0b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76102025-04-30T06:43:12.125ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
76112025-04-30T06:43:12.126ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfc290ad-cd6f-4aaf-b798-47475382c12d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76122025-04-30T06:43:12.126ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
76132025-04-30T06:43:12.127ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9868b129-de1f-47ca-bdee-4586de0ad66b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76142025-04-30T06:43:12.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
76152025-04-30T06:43:12.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5a2b6f5-00eb-4463-8aef-1cb715fd47c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76162025-04-30T06:43:12.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
76172025-04-30T06:43:12.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df694608-1df1-4a40-bfc9-39c23c994950", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76182025-04-30T06:43:12.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
76192025-04-30T06:43:12.134ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a67c9b5-53ed-4907-b14d-ffe76e53917f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76202025-04-30T06:43:12.134ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
76212025-04-30T06:43:12.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cb259ba-cfb3-491f-b285-846fd1876c23", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76222025-04-30T06:43:12.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
76232025-04-30T06:43:12.139ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c64fcfa6-5bfe-4063-a6eb-d99f2d734519", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76242025-04-30T06:43:12.139ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
76252025-04-30T06:43:12.144ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f211d63c-1535-415b-974c-6a1709d6bf7e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76262025-04-30T06:43:12.144ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
76272025-04-30T06:43:12.146ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "093f415d-10a1-4167-93c8-fa09110000d1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76282025-04-30T06:43:12.146ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
76292025-04-30T06:43:12.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52aef285-a88a-4c68-a89a-468ff9e10447", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76302025-04-30T06:43:12.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
76312025-04-30T06:43:12.152ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b71bd380-6760-4bea-b00f-263b2eec4459", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76322025-04-30T06:43:12.152ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
76332025-04-30T06:43:12.154ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3008ecbc-c6bc-4b27-95da-3f1e77355714", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76342025-04-30T06:43:12.154ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
76352025-04-30T06:43:12.157ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10e31423-c6cd-43fa-8a93-00c1a2027b64", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76362025-04-30T06:43:12.157ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
76372025-04-30T06:43:12.160ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a90fb4d-44e3-49e9-a51c-ff43e6b3e1da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76382025-04-30T06:43:12.165ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
76392025-04-30T06:43:12.166ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76402025-04-30T06:43:12.167ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76412025-04-30T06:43:12.167ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76422025-04-30T06:43:12.168ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
76432025-04-30T06:43:12.169ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76442025-04-30T06:43:12.173ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aecfe627-a867-4761-b2a2-b559db5f98a0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76452025-04-30T06:43:12.173ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76462025-04-30T06:43:12.173ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76472025-04-30T06:43:12.173ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76482025-04-30T06:43:12.173ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76492025-04-30T06:43:12.174ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76502025-04-30T06:43:12.175ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76512025-04-30T06:43:12.181ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76522025-04-30T06:43:12.182ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76532025-04-30T06:43:12.182ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
76542025-04-30T06:43:12.183ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76552025-04-30T06:43:12.183ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76562025-04-30T06:43:12.189ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76572025-04-30T06:43:12.189ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76582025-04-30T06:43:12.189ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76592025-04-30T06:43:12.189ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76602025-04-30T06:43:12.189ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76612025-04-30T06:43:12.189ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76622025-04-30T06:43:12.194ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76632025-04-30T06:43:12.194ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
76642025-04-30T06:43:12.195ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76652025-04-30T06:43:12.196ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76662025-04-30T06:43:12.205ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76672025-04-30T06:43:12.205ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76682025-04-30T06:43:12.205ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76692025-04-30T06:43:12.206ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76702025-04-30T06:43:12.284ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
76712025-04-30T06:43:12.284ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
76722025-04-30T06:43:12.285ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:40027 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
76732025-04-30T06:43:12.286ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:40027 req_id = 2262cd8b-4bac-4a49-b3fb-54f300a42e75 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76742025-04-30T06:43:12.287ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 573 local_addr = [::1]:51119 method = GET remote_addr = [::1]:40027 req_id = 2262cd8b-4bac-4a49-b3fb-54f300a42e75 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76752025-04-30T06:43:12.287ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2262cd8b-4bac-4a49-b3fb-54f300a42e75", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
76762025-04-30T06:43:12.289ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76772025-04-30T06:43:12.289ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76782025-04-30T06:43:12.289ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
76792025-04-30T06:43:12.290ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76802025-04-30T06:43:12.291ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76812025-04-30T06:43:12.291ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76822025-04-30T06:43:12.291ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76832025-04-30T06:43:12.291ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76842025-04-30T06:43:12.295ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76852025-04-30T06:43:12.300ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
76862025-04-30T06:43:12.300ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76872025-04-30T06:43:12.300ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76882025-04-30T06:43:12.301ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76892025-04-30T06:43:12.323ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
76902025-04-30T06:43:12.336ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
76912025-04-30T06:43:12.337ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
76922025-04-30T06:43:12.337ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 530.390779ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
76932025-04-30T06:43:12.338ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
76942025-04-30T06:43:12.338ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76952025-04-30T06:43:12.338ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76962025-04-30T06:43:12.338ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76972025-04-30T06:43:12.338ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76982025-04-30T06:43:12.339ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76992025-04-30T06:43:12.341ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77002025-04-30T06:43:12.355ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77012025-04-30T06:43:12.355ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77022025-04-30T06:43:12.355ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77032025-04-30T06:43:12.356ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77042025-04-30T06:43:12.362ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
77052025-04-30T06:43:12.362ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
77062025-04-30T06:43:12.364ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1e 26 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .&..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63479 req_id = ffdd3461-5471-409e-b35b-a61fcdc0c0be
77072025-04-30T06:43:12.364ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7718,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63479 req_id = ffdd3461-5471-409e-b35b-a61fcdc0c0be
77082025-04-30T06:43:12.364ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
77092025-04-30T06:43:12.365ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77102025-04-30T06:43:12.365ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63479 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ffdd3461-5471-409e-b35b-a61fcdc0c0be
77112025-04-30T06:43:12.366ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4257, tv_nsec: 164970245 } })
77122025-04-30T06:43:12.366ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
77132025-04-30T06:43:12.366ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 19 7c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .|..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49415 req_id = 5c2d57e0-f2b5-4954-b364-531839b81ef9
77142025-04-30T06:43:12.366ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6524,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49415 req_id = 5c2d57e0-f2b5-4954-b364-531839b81ef9
77152025-04-30T06:43:12.367ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77162025-04-30T06:43:12.367ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49415 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5c2d57e0-f2b5-4954-b364-531839b81ef9
77172025-04-30T06:43:12.442ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
77182025-04-30T06:43:12.443ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77192025-04-30T06:43:12.443ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77202025-04-30T06:43:12.443ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77212025-04-30T06:43:12.443ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77222025-04-30T06:43:12.444ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77232025-04-30T06:43:12.444ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c1 90 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50328 req_id = 23fb5659-cf6b-47d0-9970-1752908e540a
77242025-04-30T06:43:12.444ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49552,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50328 req_id = 23fb5659-cf6b-47d0-9970-1752908e540a
77252025-04-30T06:43:12.444ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
77262025-04-30T06:43:12.444ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49552, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50328 req_id = 23fb5659-cf6b-47d0-9970-1752908e540a
77272025-04-30T06:43:12.445ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
77282025-04-30T06:43:12.445ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
77292025-04-30T06:43:12.445ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
77302025-04-30T06:43:12.447ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "df70217e-4c36-45c7-b44f-c793afe8268e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77312025-04-30T06:43:12.447ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
77322025-04-30T06:43:12.447ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77332025-04-30T06:43:12.454ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77342025-04-30T06:43:12.454ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77352025-04-30T06:43:12.455ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77362025-04-30T06:43:12.455ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77372025-04-30T06:43:12.458ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee045af0-3118-4940-b572-b48852dd8073", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77382025-04-30T06:43:12.459ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
77392025-04-30T06:43:12.459ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77402025-04-30T06:43:12.459ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77412025-04-30T06:43:12.466ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77422025-04-30T06:43:12.473ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8ccde1f-f6f4-4eed-85e1-0060a95d8aaa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77432025-04-30T06:43:12.473ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
77442025-04-30T06:43:12.476ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42dc2aa7-b48b-4d60-9980-5d8e7718e536", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77452025-04-30T06:43:12.476ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
77462025-04-30T06:43:12.479ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62d91557-1966-48cb-b9c0-3283f98122ea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77472025-04-30T06:43:12.479ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
77482025-04-30T06:43:12.482ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9ec4dc3-a2af-4ed3-9118-6de966406083", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77492025-04-30T06:43:12.482ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
77502025-04-30T06:43:12.491ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c241cc0-cd46-4238-ac6c-08a342e5f247", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77512025-04-30T06:43:12.491ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
77522025-04-30T06:43:12.494ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22871a3b-891b-4380-b3c8-0ecabb337078", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77532025-04-30T06:43:12.494ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
77542025-04-30T06:43:12.498ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff760459-38f8-4b5f-97c1-5a62ccc11e9e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77552025-04-30T06:43:12.498ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
77562025-04-30T06:43:12.501ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d23d0733-313f-4177-9f81-2784b51944a0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77572025-04-30T06:43:12.501ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
77582025-04-30T06:43:12.504ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67ce1804-fd92-4dd0-8142-6fd8bac98bca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77592025-04-30T06:43:12.504ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
77602025-04-30T06:43:12.508ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "879d77d5-772b-472c-b771-bd0a5ab4b765", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77612025-04-30T06:43:12.508ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
77622025-04-30T06:43:12.509ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
77632025-04-30T06:43:12.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a81d18fb-e8cd-46ec-9088-ed58bc290b9d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77642025-04-30T06:43:12.511ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
77652025-04-30T06:43:12.516ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ee824e9-621f-48ab-9981-4de681030620", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77662025-04-30T06:43:12.516ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
77672025-04-30T06:43:12.518ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cb9f9ce-0160-4ea7-9d78-c409e48eece5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77682025-04-30T06:43:12.518ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
77692025-04-30T06:43:12.524ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dbcdff1-e6cc-4ceb-a557-ad1268821ff0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77702025-04-30T06:43:12.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
77712025-04-30T06:43:12.536ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0a7b7a2-eb06-41c5-8c30-029412d52881", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77722025-04-30T06:43:12.537ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
77732025-04-30T06:43:12.542ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad0fb41-c03a-4817-9518-e7ef9aeecb28", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77742025-04-30T06:43:12.542ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
77752025-04-30T06:43:12.555ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3b5ffd5-ceb3-4de8-b954-fd9f1cc1af5a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77762025-04-30T06:43:12.562ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
77772025-04-30T06:43:12.562ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77782025-04-30T06:43:12.562ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77792025-04-30T06:43:12.563ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77802025-04-30T06:43:12.563ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77812025-04-30T06:43:12.564ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77822025-04-30T06:43:12.567ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8921b5e4-1b03-43c9-98b7-8e41da00cf33", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77832025-04-30T06:43:12.567ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
77842025-04-30T06:43:12.569ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77852025-04-30T06:43:12.578ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77862025-04-30T06:43:12.578ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77872025-04-30T06:43:12.578ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77882025-04-30T06:43:12.579ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77892025-04-30T06:43:12.579ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1224cac-d804-46df-be6c-fc5e41f8ed9d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77902025-04-30T06:43:12.580ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
77912025-04-30T06:43:12.586ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77922025-04-30T06:43:12.586ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77932025-04-30T06:43:12.587ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77942025-04-30T06:43:12.588ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d203054c-7a4e-42f2-9954-fa644e7573a2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77952025-04-30T06:43:12.589ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
77962025-04-30T06:43:12.591ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1fa6ce2-1236-4ade-9fd7-7e3896cabdfc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77972025-04-30T06:43:12.591ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
77982025-04-30T06:43:12.595ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc394c5d-45f1-4f0b-aa20-b2f862d50a08", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
77992025-04-30T06:43:12.596ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
78002025-04-30T06:43:12.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f54778ce-4681-4436-bd62-910add87bdf7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78012025-04-30T06:43:12.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
78022025-04-30T06:43:12.605ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7314c6a6-62be-4bbe-b281-9dc8276e25c0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78032025-04-30T06:43:12.605ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
78042025-04-30T06:43:12.610ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff211fff-5d2f-4096-93c6-576af699eb74", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78052025-04-30T06:43:12.610ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
78062025-04-30T06:43:12.615ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9ac28e5-bf52-4b01-b22f-79f51e5270aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78072025-04-30T06:43:12.615ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
78082025-04-30T06:43:12.627ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72467779-5879-47a4-8661-2815fbe4d94b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78092025-04-30T06:43:12.628ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
78102025-04-30T06:43:12.630ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f14a8bd8-15d4-4e9d-9924-de88fda1e147", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78112025-04-30T06:43:12.630ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
78122025-04-30T06:43:12.641ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5184646-9bbb-4e20-9fd1-1068f4ab3b9d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78132025-04-30T06:43:12.642ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
78142025-04-30T06:43:12.642ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
78152025-04-30T06:43:12.646ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53b5a9fe-aaa7-4f35-96c5-7a050d54aeff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78162025-04-30T06:43:12.646ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
78172025-04-30T06:43:12.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa0c7b2e-63b1-4432-be39-a3a436ddb581", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78182025-04-30T06:43:12.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
78192025-04-30T06:43:12.649ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "feb16f3e-3dd9-4f32-8157-c87162db973e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78202025-04-30T06:43:12.649ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
78212025-04-30T06:43:12.650ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2310d141-564c-4a61-9fb9-c8d0bd6834d6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78222025-04-30T06:43:12.651ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
78232025-04-30T06:43:12.653ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be249bee-4b7d-45de-a4c8-ffb22e6ba9e0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78242025-04-30T06:43:12.653ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
78252025-04-30T06:43:12.658ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e88fa0f5-9a3e-4e8c-95ff-faf20eee55b1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78262025-04-30T06:43:12.658ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
78272025-04-30T06:43:12.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10ce1520-027a-4e44-ae6a-a6358ccf7260", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78282025-04-30T06:43:12.662ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
78292025-04-30T06:43:12.664ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a087672e-a845-4ba1-9ed3-d91da767578b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78302025-04-30T06:43:12.664ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
78312025-04-30T06:43:12.667ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d40f094-8593-4b56-8a7b-2eff3c80b1ca", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78322025-04-30T06:43:12.667ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
78332025-04-30T06:43:12.669ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9412bff5-de56-4f5d-8a37-bcb3c751f252", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78342025-04-30T06:43:12.669ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
78352025-04-30T06:43:12.670ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72489dad-8eed-44c5-aca1-f2bf754bf8bd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78362025-04-30T06:43:12.671ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
78372025-04-30T06:43:12.672ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5e70947-ce70-413a-9125-4045f6070440", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78382025-04-30T06:43:12.672ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
78392025-04-30T06:43:12.673ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d90b6fc9-cae0-49fe-8fdf-6e4aa9bd1f59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78402025-04-30T06:43:12.673ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
78412025-04-30T06:43:12.674ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "894f03ce-73b0-410e-9d9d-f0b0084e5591", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78422025-04-30T06:43:12.674ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
78432025-04-30T06:43:12.674ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78442025-04-30T06:43:12.674ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78452025-04-30T06:43:12.675ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78462025-04-30T06:43:12.677ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78472025-04-30T06:43:12.680ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78482025-04-30T06:43:12.686ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
78492025-04-30T06:43:12.686ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
78502025-04-30T06:43:12.686ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78512025-04-30T06:43:12.686ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78522025-04-30T06:43:12.686ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78532025-04-30T06:43:12.687ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78542025-04-30T06:43:12.688ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78552025-04-30T06:43:12.688ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78562025-04-30T06:43:12.689ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
78572025-04-30T06:43:12.690ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
78582025-04-30T06:43:12.692ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78592025-04-30T06:43:12.700ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
78602025-04-30T06:43:12.700ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78612025-04-30T06:43:12.700ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78622025-04-30T06:43:12.701ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78632025-04-30T06:43:12.702ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78642025-04-30T06:43:12.703ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78652025-04-30T06:43:12.703ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78662025-04-30T06:43:12.703ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
78672025-04-30T06:43:12.704ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78682025-04-30T06:43:12.704ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
78692025-04-30T06:43:12.704ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
78702025-04-30T06:43:12.797ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
78712025-04-30T06:43:12.797ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
78722025-04-30T06:43:12.798ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:44962 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
78732025-04-30T06:43:12.799ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
78742025-04-30T06:43:12.799ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78752025-04-30T06:43:12.799ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78762025-04-30T06:43:12.799ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78772025-04-30T06:43:12.800ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
78782025-04-30T06:43:12.801ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78792025-04-30T06:43:12.811ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78802025-04-30T06:43:12.811ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78812025-04-30T06:43:12.811ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
78822025-04-30T06:43:12.811ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78832025-04-30T06:43:12.812ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:44962 req_id = 21008018-af9b-4ef1-b16b-ef24add51099 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
78842025-04-30T06:43:12.814ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 2405 local_addr = [::1]:51119 method = GET remote_addr = [::1]:44962 req_id = 21008018-af9b-4ef1-b16b-ef24add51099 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
78852025-04-30T06:43:12.816ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78862025-04-30T06:43:12.816ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78872025-04-30T06:43:12.816ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78882025-04-30T06:43:12.816ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78892025-04-30T06:43:12.816ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "21008018-af9b-4ef1-b16b-ef24add51099", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
78902025-04-30T06:43:12.819ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
78912025-04-30T06:43:12.819ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
78922025-04-30T06:43:12.819ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
78932025-04-30T06:43:12.822ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
78942025-04-30T06:43:12.822ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78952025-04-30T06:43:12.822ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78962025-04-30T06:43:12.822ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78972025-04-30T06:43:12.822ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78982025-04-30T06:43:12.823ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78992025-04-30T06:43:12.823ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
79002025-04-30T06:43:12.824ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79012025-04-30T06:43:12.827ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79022025-04-30T06:43:12.832ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79032025-04-30T06:43:12.832ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79042025-04-30T06:43:12.833ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79052025-04-30T06:43:12.834ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79062025-04-30T06:43:12.868ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
79072025-04-30T06:43:12.875ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
79082025-04-30T06:43:12.876ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
79092025-04-30T06:43:12.876ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 538.431359ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
79102025-04-30T06:43:12.876ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
79112025-04-30T06:43:12.876ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79122025-04-30T06:43:12.876ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79132025-04-30T06:43:12.876ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79142025-04-30T06:43:12.876ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79152025-04-30T06:43:12.877ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79162025-04-30T06:43:12.880ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79172025-04-30T06:43:12.886ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79182025-04-30T06:43:12.887ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79192025-04-30T06:43:12.887ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79202025-04-30T06:43:12.889ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79212025-04-30T06:43:12.893ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
79222025-04-30T06:43:12.893ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
79232025-04-30T06:43:12.893ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 05 f4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51733 req_id = b33b5e6a-14b5-4e24-bfc7-2180a557dec1
79242025-04-30T06:43:12.894ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1524,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51733 req_id = b33b5e6a-14b5-4e24-bfc7-2180a557dec1
79252025-04-30T06:43:12.894ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
79262025-04-30T06:43:12.895ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79272025-04-30T06:43:12.895ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51733 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b33b5e6a-14b5-4e24-bfc7-2180a557dec1
79282025-04-30T06:43:12.895ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4257, tv_nsec: 694467163 } })
79292025-04-30T06:43:12.895ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
79302025-04-30T06:43:12.896ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 51 7e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Q~..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62722 req_id = a15516d9-ec7e-41dd-b7d8-ee4a427bf4bb
79312025-04-30T06:43:12.896ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20862,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62722 req_id = a15516d9-ec7e-41dd-b7d8-ee4a427bf4bb
79322025-04-30T06:43:12.896ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79332025-04-30T06:43:12.896ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62722 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a15516d9-ec7e-41dd-b7d8-ee4a427bf4bb
79342025-04-30T06:43:12.949ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
79352025-04-30T06:43:12.949ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79362025-04-30T06:43:12.949ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79372025-04-30T06:43:12.949ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79382025-04-30T06:43:12.949ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
79392025-04-30T06:43:12.950ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79402025-04-30T06:43:12.951ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 60 f5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c `............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59819 req_id = 72f3da54-233f-4f88-987b-02d6cbecc99a
79412025-04-30T06:43:12.952ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24821,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59819 req_id = 72f3da54-233f-4f88-987b-02d6cbecc99a
79422025-04-30T06:43:12.952ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
79432025-04-30T06:43:12.952ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24821, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59819 req_id = 72f3da54-233f-4f88-987b-02d6cbecc99a
79442025-04-30T06:43:12.953ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
79452025-04-30T06:43:12.953ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
79462025-04-30T06:43:12.953ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
79472025-04-30T06:43:12.955ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8324ae2f-3426-42c3-a2a9-e7d3d783c656", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79482025-04-30T06:43:12.955ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
79492025-04-30T06:43:12.957ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99c26250-fdf7-496b-a2fd-5d6c2a926827", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79502025-04-30T06:43:12.957ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
79512025-04-30T06:43:12.964ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79522025-04-30T06:43:12.970ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79532025-04-30T06:43:12.971ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79542025-04-30T06:43:12.971ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
79552025-04-30T06:43:12.971ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79562025-04-30T06:43:12.974ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79572025-04-30T06:43:12.974ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
79582025-04-30T06:43:12.975ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79592025-04-30T06:43:12.975ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8a886fd-920d-48ac-89fb-da4171d9968c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79602025-04-30T06:43:12.975ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
79612025-04-30T06:43:12.980ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52535533-6590-4a59-9b9e-3dda64654d28", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79622025-04-30T06:43:12.980ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
79632025-04-30T06:43:13.044ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1304c44-0593-4ba7-96c6-9e898843bc2b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79642025-04-30T06:43:13.047ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
79652025-04-30T06:43:13.048ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
79662025-04-30T06:43:13.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43f50163-77c3-44ec-b78a-58e73cc13981", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79672025-04-30T06:43:13.056ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
79682025-04-30T06:43:13.058ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04d94aba-79ba-498d-85c3-86ea8836bd2f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79692025-04-30T06:43:13.058ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
79702025-04-30T06:43:13.061ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb8bea51-13c3-44a9-ac5c-23e6dfb0f0e5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79712025-04-30T06:43:13.061ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
79722025-04-30T06:43:13.062ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "705a3f0f-241c-48df-8cae-ba5a96de27be", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79732025-04-30T06:43:13.063ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
79742025-04-30T06:43:13.064ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "161894d2-5c27-4010-9997-b02293d8b29d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79752025-04-30T06:43:13.065ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
79762025-04-30T06:43:13.066ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a1bf6ef-dab3-42de-ab71-3456256a46d1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79772025-04-30T06:43:13.066ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
79782025-04-30T06:43:13.067ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1560fa03-756b-4798-a3f8-eb199ccb39b4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79792025-04-30T06:43:13.068ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
79802025-04-30T06:43:13.069ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1be8f95-c0ef-4fe4-a5dd-78bd3107710f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79812025-04-30T06:43:13.069ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
79822025-04-30T06:43:13.072ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1076957-27ae-4ba1-b5d3-6a0a8d92b26b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79832025-04-30T06:43:13.072ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
79842025-04-30T06:43:13.077ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26f81fad-3038-477b-b586-0d6b2c993db4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79852025-04-30T06:43:13.077ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
79862025-04-30T06:43:13.080ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "386c7f68-9141-47a0-a8e5-b9ab236b87b7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79872025-04-30T06:43:13.080ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
79882025-04-30T06:43:13.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b991522-bcef-41c2-8c21-408e30d02a35", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79892025-04-30T06:43:13.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
79902025-04-30T06:43:13.085ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed99fb5a-4286-4fd7-afc7-e0d5b26950f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79912025-04-30T06:43:13.085ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
79922025-04-30T06:43:13.087ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f13527e-4c9c-4284-9bee-fe77477e8269", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79932025-04-30T06:43:13.087ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
79942025-04-30T06:43:13.088ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06caa9a8-aa2c-4fe5-903e-ccb10d0d7cc0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79952025-04-30T06:43:13.089ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
79962025-04-30T06:43:13.090ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08d48c13-34f8-48a5-bac1-3ae5de5bed64", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79972025-04-30T06:43:13.090ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
79982025-04-30T06:43:13.091ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83875532-bcc5-4fbf-bb41-94c3e5766e02", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
79992025-04-30T06:43:13.091ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
80002025-04-30T06:43:13.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9a47db2-f7fb-4d3c-9e4d-bc2dcbe6f5bf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80012025-04-30T06:43:13.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
80022025-04-30T06:43:13.094ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "061ddb2a-0a29-42a8-a9a0-d82070506760", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80032025-04-30T06:43:13.094ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
80042025-04-30T06:43:13.096ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0cfa7ed-9802-4a6b-95ca-4997965fef68", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80052025-04-30T06:43:13.096ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
80062025-04-30T06:43:13.100ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80072025-04-30T06:43:13.100ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80082025-04-30T06:43:13.100ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80092025-04-30T06:43:13.100ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80102025-04-30T06:43:13.101ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80112025-04-30T06:43:13.102ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdc5492c-02a7-4fb2-9496-2b034aeb5ec9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80122025-04-30T06:43:13.102ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
80132025-04-30T06:43:13.103ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80142025-04-30T06:43:13.109ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80152025-04-30T06:43:13.109ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80162025-04-30T06:43:13.109ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80172025-04-30T06:43:13.111ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80182025-04-30T06:43:13.117ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80192025-04-30T06:43:13.117ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80202025-04-30T06:43:13.117ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80212025-04-30T06:43:13.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4313db0-8a26-4099-9c90-215af5cf4568", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80222025-04-30T06:43:13.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
80232025-04-30T06:43:13.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "739edaaf-3cb2-4417-adc4-260e1ea3fbe1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80242025-04-30T06:43:13.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
80252025-04-30T06:43:13.122ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29da2819-aa70-4092-a548-150d5a01fd5d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80262025-04-30T06:43:13.122ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
80272025-04-30T06:43:13.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "302991fb-38b0-48cd-8a83-1b1705e2ca13", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80282025-04-30T06:43:13.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
80292025-04-30T06:43:13.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcb7187b-7e7e-46ea-8890-b6e37cf3fda2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80302025-04-30T06:43:13.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
80312025-04-30T06:43:13.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c7496b5-f0b7-402e-939e-eaeba4db64c7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80322025-04-30T06:43:13.131ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
80332025-04-30T06:43:13.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fc42c94-0e5d-4c7e-845e-052195905a16", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80342025-04-30T06:43:13.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
80352025-04-30T06:43:13.133ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bb7d6b4-4f41-4e60-ac6c-f98b49663ab1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80362025-04-30T06:43:13.133ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
80372025-04-30T06:43:13.135ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "442b0d4b-6c8d-437f-844b-47f6a2578e52", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80382025-04-30T06:43:13.135ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
80392025-04-30T06:43:13.139ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9854579d-4fa5-482a-a92d-a082f30b1c6f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80402025-04-30T06:43:13.139ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
80412025-04-30T06:43:13.147ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "467fff05-0974-4c2e-8b0e-8804a7a5113c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80422025-04-30T06:43:13.148ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
80432025-04-30T06:43:13.148ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
80442025-04-30T06:43:13.152ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6b4c9c9-cda5-4086-b896-c2e09bb29377", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80452025-04-30T06:43:13.152ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
80462025-04-30T06:43:13.155ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee2bf3b5-a349-4e46-8801-feac5d3aab3b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80472025-04-30T06:43:13.156ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
80482025-04-30T06:43:13.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9878cfea-a592-4b00-a49d-5c601fde068f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80492025-04-30T06:43:13.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
80502025-04-30T06:43:13.161ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1457cc63-45d4-40a2-a303-e98fa014fb9e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80512025-04-30T06:43:13.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
80522025-04-30T06:43:13.163ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34d97a49-f21c-4474-a32e-30e3776eaead", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80532025-04-30T06:43:13.163ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
80542025-04-30T06:43:13.167ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83da8896-09ad-4aec-85b3-18528a607a07", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80552025-04-30T06:43:13.167ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
80562025-04-30T06:43:13.168ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f32cef0a-d6d5-45ee-8f8b-74d4f9d3772a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80572025-04-30T06:43:13.168ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
80582025-04-30T06:43:13.169ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12f1b95f-adfa-4e4a-adad-0cc69d7a2514", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80592025-04-30T06:43:13.170ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
80602025-04-30T06:43:13.170ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80612025-04-30T06:43:13.170ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80622025-04-30T06:43:13.170ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80632025-04-30T06:43:13.172ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80642025-04-30T06:43:13.175ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80652025-04-30T06:43:13.179ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80662025-04-30T06:43:13.179ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
80672025-04-30T06:43:13.179ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80682025-04-30T06:43:13.179ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80692025-04-30T06:43:13.179ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80702025-04-30T06:43:13.179ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80712025-04-30T06:43:13.183ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80722025-04-30T06:43:13.194ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80732025-04-30T06:43:13.194ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80742025-04-30T06:43:13.194ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80752025-04-30T06:43:13.195ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80762025-04-30T06:43:13.200ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80772025-04-30T06:43:13.200ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80782025-04-30T06:43:13.200ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80792025-04-30T06:43:13.200ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80802025-04-30T06:43:13.200ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80812025-04-30T06:43:13.263ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
80822025-04-30T06:43:13.263ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
80832025-04-30T06:43:13.264ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:41834 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
80842025-04-30T06:43:13.265ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:41834 req_id = b76cf3ff-81d3-4778-977b-55344d6a8b87 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
80852025-04-30T06:43:13.265ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 550 local_addr = [::1]:51119 method = GET remote_addr = [::1]:41834 req_id = b76cf3ff-81d3-4778-977b-55344d6a8b87 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
80862025-04-30T06:43:13.266ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80872025-04-30T06:43:13.285ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80882025-04-30T06:43:13.285ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80892025-04-30T06:43:13.285ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80902025-04-30T06:43:13.286ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80912025-04-30T06:43:13.286ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b76cf3ff-81d3-4778-977b-55344d6a8b87", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
80922025-04-30T06:43:13.289ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
80932025-04-30T06:43:13.289ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
80942025-04-30T06:43:13.289ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
80952025-04-30T06:43:13.292ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
80962025-04-30T06:43:13.292ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80972025-04-30T06:43:13.293ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80982025-04-30T06:43:13.293ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80992025-04-30T06:43:13.293ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81002025-04-30T06:43:13.296ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81012025-04-30T06:43:13.296ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
81022025-04-30T06:43:13.297ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81032025-04-30T06:43:13.298ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81042025-04-30T06:43:13.306ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81052025-04-30T06:43:13.308ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81062025-04-30T06:43:13.308ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81072025-04-30T06:43:13.309ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81082025-04-30T06:43:13.359ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
81092025-04-30T06:43:13.360ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
81102025-04-30T06:43:13.360ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 484.210267ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
81112025-04-30T06:43:13.360ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
81122025-04-30T06:43:13.361ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81132025-04-30T06:43:13.361ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81142025-04-30T06:43:13.361ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81152025-04-30T06:43:13.361ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81162025-04-30T06:43:13.363ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81172025-04-30T06:43:13.363ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
81182025-04-30T06:43:13.375ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81192025-04-30T06:43:13.379ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81202025-04-30T06:43:13.379ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81212025-04-30T06:43:13.379ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81222025-04-30T06:43:13.380ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81232025-04-30T06:43:13.388ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
81242025-04-30T06:43:13.388ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
81252025-04-30T06:43:13.389ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 85 a8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64728 req_id = 95f73a09-a556-4d97-b9c1-c738bd4de88b
81262025-04-30T06:43:13.389ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34216,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64728 req_id = 95f73a09-a556-4d97-b9c1-c738bd4de88b
81272025-04-30T06:43:13.390ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
81282025-04-30T06:43:13.390ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
81292025-04-30T06:43:13.390ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64728 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 95f73a09-a556-4d97-b9c1-c738bd4de88b
81302025-04-30T06:43:13.391ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4258, tv_nsec: 189941934 } })
81312025-04-30T06:43:13.391ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
81322025-04-30T06:43:13.391ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 44 b2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e D...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58156 req_id = 60ffdbf5-4e98-4537-9dc9-f1b8b5e00b6f
81332025-04-30T06:43:13.391ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17586,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58156 req_id = 60ffdbf5-4e98-4537-9dc9-f1b8b5e00b6f
81342025-04-30T06:43:13.392ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
81352025-04-30T06:43:13.392ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58156 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 60ffdbf5-4e98-4537-9dc9-f1b8b5e00b6f
81362025-04-30T06:43:13.461ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
81372025-04-30T06:43:13.461ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81382025-04-30T06:43:13.461ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81392025-04-30T06:43:13.461ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81402025-04-30T06:43:13.461ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
81412025-04-30T06:43:13.462ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81422025-04-30T06:43:13.462ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7f 3e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .>..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55090 req_id = 9e4e7a54-28ad-4c9a-808d-6550ec2f731e
81432025-04-30T06:43:13.462ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32574,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55090 req_id = 9e4e7a54-28ad-4c9a-808d-6550ec2f731e
81442025-04-30T06:43:13.463ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
81452025-04-30T06:43:13.463ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32574, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55090 req_id = 9e4e7a54-28ad-4c9a-808d-6550ec2f731e
81462025-04-30T06:43:13.463ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
81472025-04-30T06:43:13.463ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
81482025-04-30T06:43:13.463ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
81492025-04-30T06:43:13.475ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81502025-04-30T06:43:13.481ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81512025-04-30T06:43:13.481ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81522025-04-30T06:43:13.482ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
81532025-04-30T06:43:13.482ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81542025-04-30T06:43:13.485ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "81ccaf18-1419-4aae-a8f0-83697591b2f8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81552025-04-30T06:43:13.485ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
81562025-04-30T06:43:13.486ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81572025-04-30T06:43:13.486ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
81582025-04-30T06:43:13.486ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81592025-04-30T06:43:13.489ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e226657e-3a93-410d-991b-fa00531149b2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81602025-04-30T06:43:13.489ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
81612025-04-30T06:43:13.492ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01b67b3d-24c7-410f-ba88-fd28f078ce5b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81622025-04-30T06:43:13.492ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
81632025-04-30T06:43:13.496ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18a2722b-7768-461e-a974-5d98d6a938a6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81642025-04-30T06:43:13.496ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
81652025-04-30T06:43:13.506ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb578771-9187-4d93-8822-b9ff6276d913", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81662025-04-30T06:43:13.506ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
81672025-04-30T06:43:13.509ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "703169d5-2605-4ce2-bcfa-5d23d0b4d446", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81682025-04-30T06:43:13.509ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
81692025-04-30T06:43:13.511ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd1f5ef2-457f-46b2-83d7-de2567f307c5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81702025-04-30T06:43:13.511ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
81712025-04-30T06:43:13.514ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbfc5418-a733-44d2-8d53-db93298350ea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81722025-04-30T06:43:13.514ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
81732025-04-30T06:43:13.515ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2de41ac9-c145-411d-acbf-17fd99d4dcaf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81742025-04-30T06:43:13.515ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
81752025-04-30T06:43:13.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be473d21-6492-41c4-abb4-a3f7759ec264", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81762025-04-30T06:43:13.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
81772025-04-30T06:43:13.521ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "186d5ed6-77fe-4346-8a60-78a608bf18eb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81782025-04-30T06:43:13.521ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
81792025-04-30T06:43:13.523ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be863026-0d76-4256-97b9-54a3b686b532", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81802025-04-30T06:43:13.523ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
81812025-04-30T06:43:13.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0120419b-0440-4132-b2e9-16f06197ed50", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81822025-04-30T06:43:13.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
81832025-04-30T06:43:13.531ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
81842025-04-30T06:43:13.539ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad86de0-2c7f-4d22-9e8a-9753351646c9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81852025-04-30T06:43:13.540ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
81862025-04-30T06:43:13.543ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37aefd56-ad1f-42bd-ae92-33c1a3f2cf86", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81872025-04-30T06:43:13.543ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
81882025-04-30T06:43:13.545ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3c37ceb-c2d4-4b1d-87e2-b9ce2266c6e7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81892025-04-30T06:43:13.545ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
81902025-04-30T06:43:13.548ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "221d4ad3-905b-4a46-8fbe-edcc7d56d226", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81912025-04-30T06:43:13.549ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
81922025-04-30T06:43:13.551ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78642d42-5ebd-4ad5-999b-1c1f9ecd9d3a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81932025-04-30T06:43:13.552ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
81942025-04-30T06:43:13.555ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40ce60d3-efca-4613-a0b2-438a7f4d28e8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81952025-04-30T06:43:13.555ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
81962025-04-30T06:43:13.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "073d8751-ffef-4535-b437-0a19632b5d10", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81972025-04-30T06:43:13.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
81982025-04-30T06:43:13.559ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "618ee1f4-c00b-4ee5-b801-7c0b5fcfa24e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
81992025-04-30T06:43:13.559ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
82002025-04-30T06:43:13.561ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "726208ed-0158-4b4d-9227-ee05a75c87ed", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82012025-04-30T06:43:13.562ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
82022025-04-30T06:43:13.563ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10c3b72-e1d9-47de-b9fc-7be94bae4dd0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82032025-04-30T06:43:13.563ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
82042025-04-30T06:43:13.566ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41aad9d4-a69f-49b7-9953-77cbf3b83abe", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82052025-04-30T06:43:13.566ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
82062025-04-30T06:43:13.569ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bafefcc4-faca-4441-aea3-70c6fa7cfabf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82072025-04-30T06:43:13.569ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
82082025-04-30T06:43:13.573ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8985031e-656c-40d2-81d1-496e14c2559c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82092025-04-30T06:43:13.573ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
82102025-04-30T06:43:13.582ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82112025-04-30T06:43:13.582ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82122025-04-30T06:43:13.582ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82132025-04-30T06:43:13.582ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
82142025-04-30T06:43:13.586ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82152025-04-30T06:43:13.586ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
82162025-04-30T06:43:13.588ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82172025-04-30T06:43:13.588ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 63bcf8b8-e641-485b-9495-02b05df79da4 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82182025-04-30T06:43:13.588ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 63bcf8b8-e641-485b-9495-02b05df79da4 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82192025-04-30T06:43:13.588ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 63bcf8b8-e641-485b-9495-02b05df79da4 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82202025-04-30T06:43:13.589ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 63bcf8b8-e641-485b-9495-02b05df79da4 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82212025-04-30T06:43:13.592ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82222025-04-30T06:43:13.600ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82232025-04-30T06:43:13.600ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82242025-04-30T06:43:13.601ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
82252025-04-30T06:43:13.601ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82262025-04-30T06:43:13.602ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 14687 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 63bcf8b8-e641-485b-9495-02b05df79da4 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82272025-04-30T06:43:13.604ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "63bcf8b8-e641-485b-9495-02b05df79da4", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82282025-04-30T06:43:13.604ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82292025-04-30T06:43:13.605ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82302025-04-30T06:43:13.605ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
82312025-04-30T06:43:13.606ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82322025-04-30T06:43:13.606ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3c8fee2a-ecdc-41eb-b15d-7f6453af23e0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82332025-04-30T06:43:13.606ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3c8fee2a-ecdc-41eb-b15d-7f6453af23e0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82342025-04-30T06:43:13.606ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3c8fee2a-ecdc-41eb-b15d-7f6453af23e0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82352025-04-30T06:43:13.607ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3c8fee2a-ecdc-41eb-b15d-7f6453af23e0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82362025-04-30T06:43:13.608ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "758aa22e-a1fa-42fa-9518-bd5a6a33f968", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82372025-04-30T06:43:13.609ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
82382025-04-30T06:43:13.621ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 14975 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3c8fee2a-ecdc-41eb-b15d-7f6453af23e0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82392025-04-30T06:43:13.625ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c8fee2a-ecdc-41eb-b15d-7f6453af23e0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82402025-04-30T06:43:13.625ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
82412025-04-30T06:43:13.625ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82422025-04-30T06:43:13.625ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
82432025-04-30T06:43:13.635ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b8de103-75b6-4858-a5fd-6493c2e48621", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82442025-04-30T06:43:13.636ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
82452025-04-30T06:43:13.643ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d35e72b7-990d-4c59-8de5-81f946c38a07", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82462025-04-30T06:43:13.643ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
82472025-04-30T06:43:13.652ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9076d8-7508-43b5-86be-6dc4dded14cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82482025-04-30T06:43:13.652ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
82492025-04-30T06:43:13.654ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3f3388e-a7e2-4e2a-80de-42885f971387", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82502025-04-30T06:43:13.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
82512025-04-30T06:43:13.658ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
82522025-04-30T06:43:13.660ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3166d528-f4d8-4407-9ffb-18eea56062fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82532025-04-30T06:43:13.660ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
82542025-04-30T06:43:13.662ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c1b80b6-ef2a-444c-b374-65eee9949119", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82552025-04-30T06:43:13.662ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
82562025-04-30T06:43:13.665ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1839cb0-8253-4f87-b322-d9c0b4c25eb3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82572025-04-30T06:43:13.665ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
82582025-04-30T06:43:13.668ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39926a4c-7b10-4388-8a92-c37bc2a3cf7a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82592025-04-30T06:43:13.668ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
82602025-04-30T06:43:13.673ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c955c2e-5245-43a7-b296-7e59cdd18a3f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82612025-04-30T06:43:13.673ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
82622025-04-30T06:43:13.676ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c19263a-7e8d-44f3-9a61-da4b2db65e7e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82632025-04-30T06:43:13.676ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
82642025-04-30T06:43:13.678ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b84c1b40-1941-4dc2-835c-c8903f733850", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82652025-04-30T06:43:13.678ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
82662025-04-30T06:43:13.680ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b9c85a0-2b3d-4de6-9266-aba26f324773", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82672025-04-30T06:43:13.681ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
82682025-04-30T06:43:13.682ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82692025-04-30T06:43:13.682ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82702025-04-30T06:43:13.682ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
82712025-04-30T06:43:13.684ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
82722025-04-30T06:43:13.686ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a78b2cec-bc38-4c57-8125-f98612e3c5a9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82732025-04-30T06:43:13.686ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
82742025-04-30T06:43:13.690ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
82752025-04-30T06:43:13.690ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
82762025-04-30T06:43:13.690ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bfcfdbb-d11d-4201-863b-bf98c90e969d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82772025-04-30T06:43:13.695ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
82782025-04-30T06:43:13.696ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b56d8b8-9d34-4970-b0ca-2eac976e2274", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82792025-04-30T06:43:13.696ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
82802025-04-30T06:43:13.696ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d35e48d-7961-4219-a6a4-c99bcc590d94", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82812025-04-30T06:43:13.696ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
82822025-04-30T06:43:13.696ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
82832025-04-30T06:43:13.697ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82842025-04-30T06:43:13.697ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06af9976-ed0d-4f55-86f4-8bfe0385d7bd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82852025-04-30T06:43:13.697ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
82862025-04-30T06:43:13.699ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f718367-6e5a-44eb-83a1-92c893536deb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
82872025-04-30T06:43:13.699ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
82882025-04-30T06:43:13.699ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82892025-04-30T06:43:13.699ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82902025-04-30T06:43:13.699ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82912025-04-30T06:43:13.701ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82922025-04-30T06:43:13.704ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82932025-04-30T06:43:13.715ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
82942025-04-30T06:43:13.715ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
82952025-04-30T06:43:13.715ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82962025-04-30T06:43:13.716ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82972025-04-30T06:43:13.716ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82982025-04-30T06:43:13.718ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82992025-04-30T06:43:13.719ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83002025-04-30T06:43:13.719ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83012025-04-30T06:43:13.719ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83022025-04-30T06:43:13.719ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83032025-04-30T06:43:13.720ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83042025-04-30T06:43:13.722ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83052025-04-30T06:43:13.727ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83062025-04-30T06:43:13.727ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83072025-04-30T06:43:13.727ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83082025-04-30T06:43:13.727ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83092025-04-30T06:43:13.729ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83102025-04-30T06:43:13.741ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83112025-04-30T06:43:13.741ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83122025-04-30T06:43:13.741ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83132025-04-30T06:43:13.741ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83142025-04-30T06:43:13.742ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83152025-04-30T06:43:13.742ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83162025-04-30T06:43:13.819ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
83172025-04-30T06:43:13.819ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
83182025-04-30T06:43:13.819ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:64960 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
83192025-04-30T06:43:13.820ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83202025-04-30T06:43:13.821ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83212025-04-30T06:43:13.821ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83222025-04-30T06:43:13.821ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83232025-04-30T06:43:13.821ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
83242025-04-30T06:43:13.822ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83252025-04-30T06:43:13.822ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83262025-04-30T06:43:13.823ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83272025-04-30T06:43:13.823ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:64960 req_id = 5e21ef74-dd43-40eb-8867-4f354aabe6d3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83282025-04-30T06:43:13.824ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 458 local_addr = [::1]:51119 method = GET remote_addr = [::1]:64960 req_id = 5e21ef74-dd43-40eb-8867-4f354aabe6d3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83292025-04-30T06:43:13.824ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e21ef74-dd43-40eb-8867-4f354aabe6d3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
83302025-04-30T06:43:13.826ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83312025-04-30T06:43:13.826ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83322025-04-30T06:43:13.826ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
83332025-04-30T06:43:13.829ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83342025-04-30T06:43:13.829ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83352025-04-30T06:43:13.829ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83362025-04-30T06:43:13.829ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83372025-04-30T06:43:13.830ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83382025-04-30T06:43:13.834ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83392025-04-30T06:43:13.843ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
83402025-04-30T06:43:13.843ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83412025-04-30T06:43:13.843ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83422025-04-30T06:43:13.844ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83432025-04-30T06:43:13.881ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
83442025-04-30T06:43:13.882ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
83452025-04-30T06:43:13.882ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 521.542109ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
83462025-04-30T06:43:13.883ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
83472025-04-30T06:43:13.883ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83482025-04-30T06:43:13.884ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83492025-04-30T06:43:13.884ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83502025-04-30T06:43:13.884ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83512025-04-30T06:43:13.886ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83522025-04-30T06:43:13.895ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83532025-04-30T06:43:13.902ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
83542025-04-30T06:43:13.902ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83552025-04-30T06:43:13.902ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83562025-04-30T06:43:13.903ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83572025-04-30T06:43:13.904ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
83582025-04-30T06:43:13.911ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
83592025-04-30T06:43:13.913ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
83602025-04-30T06:43:13.914ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: da d8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63255 req_id = 58c7d5f5-63c7-48b7-88bb-b28347b98e69
83612025-04-30T06:43:13.931ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56024,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63255 req_id = 58c7d5f5-63c7-48b7-88bb-b28347b98e69
83622025-04-30T06:43:13.931ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
83632025-04-30T06:43:13.931ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
83642025-04-30T06:43:13.931ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63255 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 58c7d5f5-63c7-48b7-88bb-b28347b98e69
83652025-04-30T06:43:13.937ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4258, tv_nsec: 731289865 } })
83662025-04-30T06:43:13.937ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
83672025-04-30T06:43:13.948ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 28 f1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e (...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57448 req_id = 489643f6-88fb-45f1-82c6-e2ec6060dfbe
83682025-04-30T06:43:13.949ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10481,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57448 req_id = 489643f6-88fb-45f1-82c6-e2ec6060dfbe
83692025-04-30T06:43:13.956ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
83702025-04-30T06:43:13.956ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57448 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 489643f6-88fb-45f1-82c6-e2ec6060dfbe
83712025-04-30T06:43:14.019ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
83722025-04-30T06:43:14.019ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83732025-04-30T06:43:14.019ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83742025-04-30T06:43:14.019ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83752025-04-30T06:43:14.019ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83762025-04-30T06:43:14.021ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83772025-04-30T06:43:14.022ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1f db 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60983 req_id = edd6d401-5c0c-4c4d-b00a-829d14508961
83782025-04-30T06:43:14.022ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8155,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60983 req_id = edd6d401-5c0c-4c4d-b00a-829d14508961
83792025-04-30T06:43:14.022ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
83802025-04-30T06:43:14.022ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8155, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60983 req_id = edd6d401-5c0c-4c4d-b00a-829d14508961
83812025-04-30T06:43:14.024ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
83822025-04-30T06:43:14.024ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
83832025-04-30T06:43:14.024ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
83842025-04-30T06:43:14.024ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83852025-04-30T06:43:14.031ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83862025-04-30T06:43:14.031ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83872025-04-30T06:43:14.031ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83882025-04-30T06:43:14.031ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83892025-04-30T06:43:14.041ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83902025-04-30T06:43:14.041ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83912025-04-30T06:43:14.041ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83922025-04-30T06:43:14.044ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd1d6012-619c-46f3-8255-f6b527b7df0d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
83932025-04-30T06:43:14.044ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
83942025-04-30T06:43:14.064ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76204345-5ded-454b-8f1f-6eb99135b05e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
83952025-04-30T06:43:14.065ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
83962025-04-30T06:43:14.084ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f2709d5-622d-44d2-b6ca-73e194af13d8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
83972025-04-30T06:43:14.084ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
83982025-04-30T06:43:14.090ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c798052-f6be-4d0a-9cab-f17f8fb6f366", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
83992025-04-30T06:43:14.090ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
84002025-04-30T06:43:14.094ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30613bcc-5956-4098-99a1-62a9f97e16a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84012025-04-30T06:43:14.094ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
84022025-04-30T06:43:14.104ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9373aff-e8a6-464f-88b3-8d187b61c9f1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84032025-04-30T06:43:14.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
84042025-04-30T06:43:14.107ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
84052025-04-30T06:43:14.112ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbe44bd3-0c0c-4a35-9e2a-9a93fb53a330", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84062025-04-30T06:43:14.112ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
84072025-04-30T06:43:14.117ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb6935c7-7a98-4d91-8b8d-80a6ec21d64b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84082025-04-30T06:43:14.117ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
84092025-04-30T06:43:14.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b6f4fa9-4b82-4959-a54d-31c06fe63784", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84102025-04-30T06:43:14.119ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
84112025-04-30T06:43:14.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddafcb4a-2d63-4125-a9b2-cb4ebd5441f8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84122025-04-30T06:43:14.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
84132025-04-30T06:43:14.121ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04852116-4f75-4cf9-a83f-55ad5ab997b2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84142025-04-30T06:43:14.122ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
84152025-04-30T06:43:14.122ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3180e922-f246-4fd4-b259-3a93d48706b2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84162025-04-30T06:43:14.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
84172025-04-30T06:43:14.124ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbf303ce-ccf1-4d64-b06c-aea3736f2fdd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84182025-04-30T06:43:14.124ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
84192025-04-30T06:43:14.126ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6941a14a-3fc1-415a-b71c-962fc11d45a7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84202025-04-30T06:43:14.126ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
84212025-04-30T06:43:14.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1adee2f-9400-422a-b882-c212adb94d93", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84222025-04-30T06:43:14.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
84232025-04-30T06:43:14.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f26a218a-7fb0-442a-b388-1c20b332a6a7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84242025-04-30T06:43:14.131ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
84252025-04-30T06:43:14.133ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cea0910a-73cb-4083-a66d-365ab953f5f7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84262025-04-30T06:43:14.133ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
84272025-04-30T06:43:14.141ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b96832fa-9c64-421b-89be-305933fa8926", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84282025-04-30T06:43:14.142ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
84292025-04-30T06:43:14.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc7a1b33-baa9-4240-a146-aa006bc6f516", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84302025-04-30T06:43:14.150ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
84312025-04-30T06:43:14.151ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4065eb80-17bb-4af1-a9e0-438135b2d24d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84322025-04-30T06:43:14.152ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
84332025-04-30T06:43:14.156ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6d5de02-1e5a-44ab-9173-853badb59ced", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84342025-04-30T06:43:14.156ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
84352025-04-30T06:43:14.158ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84362025-04-30T06:43:14.158ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84372025-04-30T06:43:14.158ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84382025-04-30T06:43:14.158ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
84392025-04-30T06:43:14.160ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84402025-04-30T06:43:14.160ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67063d6c-e72b-4d50-bbdd-1ed6c59c3695", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84412025-04-30T06:43:14.160ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
84422025-04-30T06:43:14.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff1ec9af-6071-402a-acc5-b66e91623170", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84432025-04-30T06:43:14.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
84442025-04-30T06:43:14.162ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84452025-04-30T06:43:14.175ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84462025-04-30T06:43:14.175ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84472025-04-30T06:43:14.175ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
84482025-04-30T06:43:14.176ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84492025-04-30T06:43:14.180ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84502025-04-30T06:43:14.180ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
84512025-04-30T06:43:14.181ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84522025-04-30T06:43:14.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f004d1d-df10-4c5e-b355-297f0d6d084d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84532025-04-30T06:43:14.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
84542025-04-30T06:43:14.191ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a903a573-ee9f-4075-82ea-c190deffffa2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84552025-04-30T06:43:14.191ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
84562025-04-30T06:43:14.195ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da26e5db-929f-45cd-99e5-cf8fb2755a33", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84572025-04-30T06:43:14.195ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
84582025-04-30T06:43:14.199ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90fcaceb-9457-470c-b532-41d8cbc260df", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84592025-04-30T06:43:14.199ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
84602025-04-30T06:43:14.202ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fc1de7b-5061-45a1-a8d1-9b894f078754", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84612025-04-30T06:43:14.203ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
84622025-04-30T06:43:14.207ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72a16b89-f871-44f2-810a-2bd386eb064b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84632025-04-30T06:43:14.207ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
84642025-04-30T06:43:14.211ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "469ad26f-e7f8-4fdb-9ed6-aa62c1db2761", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84652025-04-30T06:43:14.211ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
84662025-04-30T06:43:14.212ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2650c635-e299-4722-adef-98e9716ed2a0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84672025-04-30T06:43:14.212ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
84682025-04-30T06:43:14.214ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "386d90a8-21c0-44ce-9500-c6dfd20dd465", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84692025-04-30T06:43:14.214ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
84702025-04-30T06:43:14.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21c424e3-a1d0-4013-b01f-4029700a979a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84712025-04-30T06:43:14.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
84722025-04-30T06:43:14.219ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
84732025-04-30T06:43:14.219ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2d8023c-c9a5-4fcd-97a2-364e1080359b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84742025-04-30T06:43:14.219ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
84752025-04-30T06:43:14.222ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce75cbba-fd9a-47b2-b267-b4de5d6822f1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84762025-04-30T06:43:14.222ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
84772025-04-30T06:43:14.226ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58df0cef-a258-4086-bc99-4a799973263a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84782025-04-30T06:43:14.226ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
84792025-04-30T06:43:14.229ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15cde61c-0d03-4823-89b9-bd4a0fd4564e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84802025-04-30T06:43:14.229ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
84812025-04-30T06:43:14.233ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3af731fd-c980-4218-b508-6d81b1892f44", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84822025-04-30T06:43:14.233ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
84832025-04-30T06:43:14.237ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f95ae0e6-3aef-4b09-9ca6-c3d24ae85705", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84842025-04-30T06:43:14.237ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
84852025-04-30T06:43:14.241ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e2cca47-ca1e-41fb-85a8-dfeb8d4e01fb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84862025-04-30T06:43:14.241ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
84872025-04-30T06:43:14.244ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b5fe2bd-7417-46e7-be8c-50a9c06fcf36", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84882025-04-30T06:43:14.245ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
84892025-04-30T06:43:14.246ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f197b0ca-026a-4f55-ba4a-63e936c18e02", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84902025-04-30T06:43:14.246ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
84912025-04-30T06:43:14.247ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b9ca5fa-624c-48eb-94eb-535fb3774579", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84922025-04-30T06:43:14.247ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
84932025-04-30T06:43:14.253ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64acecb0-9977-4c4e-b125-534ecb845ec5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84942025-04-30T06:43:14.254ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
84952025-04-30T06:43:14.255ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b598c225-bd6a-4176-bec7-a9510957c498", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
84962025-04-30T06:43:14.255ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
84972025-04-30T06:43:14.255ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84982025-04-30T06:43:14.255ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84992025-04-30T06:43:14.255ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85002025-04-30T06:43:14.256ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85012025-04-30T06:43:14.260ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85022025-04-30T06:43:14.265ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85032025-04-30T06:43:14.265ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85042025-04-30T06:43:14.265ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85052025-04-30T06:43:14.265ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85062025-04-30T06:43:14.265ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85072025-04-30T06:43:14.266ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85082025-04-30T06:43:14.270ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85092025-04-30T06:43:14.270ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85102025-04-30T06:43:14.270ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85112025-04-30T06:43:14.270ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85122025-04-30T06:43:14.272ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85132025-04-30T06:43:14.272ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85142025-04-30T06:43:14.284ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85152025-04-30T06:43:14.284ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85162025-04-30T06:43:14.284ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85172025-04-30T06:43:14.284ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85182025-04-30T06:43:14.287ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85192025-04-30T06:43:14.293ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85202025-04-30T06:43:14.293ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85212025-04-30T06:43:14.293ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85222025-04-30T06:43:14.293ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85232025-04-30T06:43:14.348ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
85242025-04-30T06:43:14.348ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
85252025-04-30T06:43:14.349ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:37729 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
85262025-04-30T06:43:14.349ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:37729 req_id = 0fc88877-cb8c-4c30-8c9c-dfb77089b208 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
85272025-04-30T06:43:14.350ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 728 local_addr = [::1]:51119 method = GET remote_addr = [::1]:37729 req_id = 0fc88877-cb8c-4c30-8c9c-dfb77089b208 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
85282025-04-30T06:43:14.350ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85292025-04-30T06:43:14.350ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85302025-04-30T06:43:14.355ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85312025-04-30T06:43:14.355ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fc88877-cb8c-4c30-8c9c-dfb77089b208", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:13 GMT"} })
85322025-04-30T06:43:14.358ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
85332025-04-30T06:43:14.358ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
85342025-04-30T06:43:14.358ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
85352025-04-30T06:43:14.361ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
85362025-04-30T06:43:14.361ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85372025-04-30T06:43:14.361ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85382025-04-30T06:43:14.361ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85392025-04-30T06:43:14.362ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85402025-04-30T06:43:14.402ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85412025-04-30T06:43:14.413ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
85422025-04-30T06:43:14.413ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85432025-04-30T06:43:14.413ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85442025-04-30T06:43:14.415ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85452025-04-30T06:43:14.466ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
85462025-04-30T06:43:14.468ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
85472025-04-30T06:43:14.468ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
85482025-04-30T06:43:14.468ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 584.642583ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
85492025-04-30T06:43:14.468ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
85502025-04-30T06:43:14.468ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
85512025-04-30T06:43:14.468ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85522025-04-30T06:43:14.468ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85532025-04-30T06:43:14.468ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85542025-04-30T06:43:14.469ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85552025-04-30T06:43:14.471ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85562025-04-30T06:43:14.475ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
85572025-04-30T06:43:14.475ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85582025-04-30T06:43:14.475ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85592025-04-30T06:43:14.475ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85602025-04-30T06:43:14.478ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
85612025-04-30T06:43:14.478ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
85622025-04-30T06:43:14.478ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 10 bb 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59560 req_id = 7e8336f6-5b04-4df3-8114-455d00d3d9b3
85632025-04-30T06:43:14.479ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4283,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59560 req_id = 7e8336f6-5b04-4df3-8114-455d00d3d9b3
85642025-04-30T06:43:14.479ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
85652025-04-30T06:43:14.479ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
85662025-04-30T06:43:14.479ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59560 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7e8336f6-5b04-4df3-8114-455d00d3d9b3
85672025-04-30T06:43:14.479ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4259, tv_nsec: 278761787 } })
85682025-04-30T06:43:14.479ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
85692025-04-30T06:43:14.480ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 72 da 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e r...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59502 req_id = d8b63991-cfaf-495c-9468-ab50577d820b
85702025-04-30T06:43:14.480ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29402,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59502 req_id = d8b63991-cfaf-495c-9468-ab50577d820b
85712025-04-30T06:43:14.480ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
85722025-04-30T06:43:14.480ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59502 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d8b63991-cfaf-495c-9468-ab50577d820b
85732025-04-30T06:43:14.532ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
85742025-04-30T06:43:14.532ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85752025-04-30T06:43:14.533ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85762025-04-30T06:43:14.533ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85772025-04-30T06:43:14.533ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85782025-04-30T06:43:14.533ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85792025-04-30T06:43:14.534ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 59 da 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Y............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56572 req_id = 779795c0-f4f2-41c1-98b3-1ecf530a040d
85802025-04-30T06:43:14.534ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23002,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56572 req_id = 779795c0-f4f2-41c1-98b3-1ecf530a040d
85812025-04-30T06:43:14.534ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
85822025-04-30T06:43:14.535ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23002, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56572 req_id = 779795c0-f4f2-41c1-98b3-1ecf530a040d
85832025-04-30T06:43:14.535ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
85842025-04-30T06:43:14.537ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
85852025-04-30T06:43:14.537ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
85862025-04-30T06:43:14.537ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85872025-04-30T06:43:14.543ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85882025-04-30T06:43:14.543ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85892025-04-30T06:43:14.543ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85902025-04-30T06:43:14.544ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85912025-04-30T06:43:14.545ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfc0ca38-a72d-4079-8796-6415689235f5", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
85922025-04-30T06:43:14.546ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
85932025-04-30T06:43:14.546ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85942025-04-30T06:43:14.547ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85952025-04-30T06:43:14.547ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85962025-04-30T06:43:14.547ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38dc03fe-ddd3-44db-8947-168eb360ff18", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
85972025-04-30T06:43:14.548ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
85982025-04-30T06:43:14.549ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07201c70-260a-4284-97d0-0d024e06c9ba", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
85992025-04-30T06:43:14.549ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
86002025-04-30T06:43:14.551ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52c947ff-38d6-4b18-a091-1fac5bc8884c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86012025-04-30T06:43:14.551ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
86022025-04-30T06:43:14.553ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd3b558c-4cdc-4d84-adf7-8f96a9a1fbe9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86032025-04-30T06:43:14.553ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
86042025-04-30T06:43:14.555ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94c13ca9-6860-4f55-b571-af24e427bc1e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86052025-04-30T06:43:14.556ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
86062025-04-30T06:43:14.558ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83e1af2e-0d73-4dc5-ad79-1275e9f82ad2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86072025-04-30T06:43:14.558ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
86082025-04-30T06:43:14.560ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b0c4931-5171-44be-bf9f-5bd034a1cc6a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86092025-04-30T06:43:14.560ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
86102025-04-30T06:43:14.561ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4411ba0a-3b93-4347-9845-48402cf0c94f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86112025-04-30T06:43:14.561ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
86122025-04-30T06:43:14.562ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d21a118-815a-4b60-bf3e-90c5cf619a7f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86132025-04-30T06:43:14.562ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
86142025-04-30T06:43:14.564ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd8dcf24-12e3-4258-b584-5cd279659f5d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86152025-04-30T06:43:14.564ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
86162025-04-30T06:43:14.565ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6bd6f53-7ce4-40b8-8123-b44d5097fc50", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86172025-04-30T06:43:14.566ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
86182025-04-30T06:43:14.567ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "560c569f-07b0-46af-8c5e-965f55e5bc53", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86192025-04-30T06:43:14.567ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
86202025-04-30T06:43:14.571ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "804e713d-346f-497e-abae-87382a6f82dd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86212025-04-30T06:43:14.571ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
86222025-04-30T06:43:14.573ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8021486b-4eaa-4d43-a8af-376dff8fa9d0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86232025-04-30T06:43:14.573ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
86242025-04-30T06:43:14.575ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04a47e05-7b70-40f3-9738-0dc48d6986e8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86252025-04-30T06:43:14.575ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
86262025-04-30T06:43:14.585ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25bf47d0-fdf6-4aea-a4f1-11a852ccb75f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86272025-04-30T06:43:14.586ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
86282025-04-30T06:43:14.588ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a79dfe3-af97-49a0-aff9-238d2feff355", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86292025-04-30T06:43:14.588ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
86302025-04-30T06:43:14.590ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8acc1e71-1829-4f72-a49f-3ad77a7fb8d3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86312025-04-30T06:43:14.591ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
86322025-04-30T06:43:14.593ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec2ea14d-676a-40d7-a89f-2fb77824b96f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86332025-04-30T06:43:14.593ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
86342025-04-30T06:43:14.594ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
86352025-04-30T06:43:14.598ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5896ca1a-e729-488e-ac1b-9fa95beec963", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86362025-04-30T06:43:14.598ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
86372025-04-30T06:43:14.599ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bc3f99f-cfbc-4809-8976-a8a32379caa2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86382025-04-30T06:43:14.599ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
86392025-04-30T06:43:14.600ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9d5ff9b-5b40-4e66-8b56-6df9c1c9b32e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86402025-04-30T06:43:14.600ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
86412025-04-30T06:43:14.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cf5e917-0ad5-416c-aa17-951e3c04b4e8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86422025-04-30T06:43:14.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
86432025-04-30T06:43:14.604ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c139b58-896d-429c-a37f-cd512566c690", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86442025-04-30T06:43:14.604ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
86452025-04-30T06:43:14.607ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9453cc92-f681-4790-8d9f-f4b150dd0773", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86462025-04-30T06:43:14.607ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
86472025-04-30T06:43:14.610ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6fb491b-d926-4489-98eb-769751fcc41e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86482025-04-30T06:43:14.610ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
86492025-04-30T06:43:14.613ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40587def-8c81-447f-bf6e-15a6842e2fed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86502025-04-30T06:43:14.613ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
86512025-04-30T06:43:14.614ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6dbb6b1-b001-4c72-adc5-248621b8f7db", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86522025-04-30T06:43:14.614ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
86532025-04-30T06:43:14.617ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81af202c-c4de-4c8a-a6bc-5ba4f54beed8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86542025-04-30T06:43:14.617ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
86552025-04-30T06:43:14.618ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b3f10df-1506-42b2-8fca-3ac506a94228", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86562025-04-30T06:43:14.618ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
86572025-04-30T06:43:14.619ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f83acac0-4358-42e2-b120-a54c5cd46557", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86582025-04-30T06:43:14.620ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
86592025-04-30T06:43:14.621ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "635b2190-f82f-4ecc-9271-a34243f37c4e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86602025-04-30T06:43:14.621ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
86612025-04-30T06:43:14.622ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da04476e-d9e7-4006-9974-a5bb6c266f8f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86622025-04-30T06:43:14.622ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
86632025-04-30T06:43:14.624ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8e3c0cb-6755-408c-85fe-9e516c03a34f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86642025-04-30T06:43:14.624ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
86652025-04-30T06:43:14.628ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8114dd6d-0dac-4283-9507-725cfc88d0c2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86662025-04-30T06:43:14.629ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
86672025-04-30T06:43:14.630ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8ca7517-7c33-4bc0-ab1f-0fb95fd44baa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86682025-04-30T06:43:14.631ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
86692025-04-30T06:43:14.632ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2f328c7-9f6d-4955-8a73-95c0f132378f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86702025-04-30T06:43:14.632ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
86712025-04-30T06:43:14.635ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "245683ed-a6c4-4ba5-9e10-4ca8d81c20c2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86722025-04-30T06:43:14.635ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
86732025-04-30T06:43:14.645ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "626809ae-5494-4dd4-a5de-84de5b71be57", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86742025-04-30T06:43:14.645ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
86752025-04-30T06:43:14.645ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86762025-04-30T06:43:14.647ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86772025-04-30T06:43:14.647ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86782025-04-30T06:43:14.647ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
86792025-04-30T06:43:14.652ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86802025-04-30T06:43:14.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00e1001e-8699-4385-aca9-b9669c827818", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86812025-04-30T06:43:14.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
86822025-04-30T06:43:14.659ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d5014e7-6261-424b-ae85-829f287b3019", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86832025-04-30T06:43:14.659ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
86842025-04-30T06:43:14.660ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfb4f72e-e71b-4b26-b6f3-59387a87643f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86852025-04-30T06:43:14.660ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
86862025-04-30T06:43:14.660ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86872025-04-30T06:43:14.675ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86882025-04-30T06:43:14.675ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86892025-04-30T06:43:14.675ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
86902025-04-30T06:43:14.676ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86912025-04-30T06:43:14.680ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58f5d7a7-d4da-4b05-b2d6-bd693058239f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
86922025-04-30T06:43:14.680ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
86932025-04-30T06:43:14.681ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86942025-04-30T06:43:14.681ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
86952025-04-30T06:43:14.683ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86962025-04-30T06:43:14.684ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86972025-04-30T06:43:14.684ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86982025-04-30T06:43:14.686ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
86992025-04-30T06:43:14.687ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
87002025-04-30T06:43:14.688ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10a704a2-b4ff-4a66-910c-c52fc803812e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
87012025-04-30T06:43:14.688ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
87022025-04-30T06:43:14.688ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87032025-04-30T06:43:14.688ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87042025-04-30T06:43:14.688ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87052025-04-30T06:43:14.689ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87062025-04-30T06:43:14.693ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
87072025-04-30T06:43:14.693ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
87082025-04-30T06:43:14.695ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87092025-04-30T06:43:14.703ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
87102025-04-30T06:43:14.703ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
87112025-04-30T06:43:14.703ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87122025-04-30T06:43:14.703ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87132025-04-30T06:43:14.703ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87142025-04-30T06:43:14.705ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87152025-04-30T06:43:14.706ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
87162025-04-30T06:43:14.706ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
87172025-04-30T06:43:14.708ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87182025-04-30T06:43:14.719ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
87192025-04-30T06:43:14.719ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87202025-04-30T06:43:14.719ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87212025-04-30T06:43:14.720ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87222025-04-30T06:43:14.724ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87232025-04-30T06:43:14.779ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
87242025-04-30T06:43:14.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
87252025-04-30T06:43:14.779ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87262025-04-30T06:43:14.780ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:48384 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
87272025-04-30T06:43:14.783ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:48384 req_id = 59fbcacb-bcde-4876-891d-5418f2afbaa3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
87282025-04-30T06:43:14.783ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87292025-04-30T06:43:14.783ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87302025-04-30T06:43:14.783ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87312025-04-30T06:43:14.783ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87322025-04-30T06:43:14.784ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 1341 local_addr = [::1]:51119 method = GET remote_addr = [::1]:48384 req_id = 59fbcacb-bcde-4876-891d-5418f2afbaa3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
87332025-04-30T06:43:14.784ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "59fbcacb-bcde-4876-891d-5418f2afbaa3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
87342025-04-30T06:43:14.786ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
87352025-04-30T06:43:14.786ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
87362025-04-30T06:43:14.786ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
87372025-04-30T06:43:14.788ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
87382025-04-30T06:43:14.788ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87392025-04-30T06:43:14.788ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87402025-04-30T06:43:14.788ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87412025-04-30T06:43:14.790ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87422025-04-30T06:43:14.793ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
87432025-04-30T06:43:14.796ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87442025-04-30T06:43:14.799ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
87452025-04-30T06:43:14.799ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87462025-04-30T06:43:14.799ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87472025-04-30T06:43:14.800ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87482025-04-30T06:43:14.811ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
87492025-04-30T06:43:14.830ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
87502025-04-30T06:43:14.831ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
87512025-04-30T06:43:14.831ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 363.104842ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
87522025-04-30T06:43:14.833ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
87532025-04-30T06:43:14.833ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
87542025-04-30T06:43:14.833ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87552025-04-30T06:43:14.833ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87562025-04-30T06:43:14.833ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87572025-04-30T06:43:14.834ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87582025-04-30T06:43:14.837ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87592025-04-30T06:43:14.848ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
87602025-04-30T06:43:14.849ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87612025-04-30T06:43:14.849ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87622025-04-30T06:43:14.849ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87632025-04-30T06:43:14.855ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
87642025-04-30T06:43:14.855ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
87652025-04-30T06:43:14.856ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4f ca 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 O............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64270 req_id = 2aacc667-7777-4c04-ab5b-42065ed10412
87662025-04-30T06:43:14.857ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20426,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64270 req_id = 2aacc667-7777-4c04-ab5b-42065ed10412
87672025-04-30T06:43:14.857ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
87682025-04-30T06:43:14.857ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
87692025-04-30T06:43:14.857ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64270 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2aacc667-7777-4c04-ab5b-42065ed10412
87702025-04-30T06:43:14.857ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4259, tv_nsec: 656339135 } })
87712025-04-30T06:43:14.857ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
87722025-04-30T06:43:14.860ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 09 21 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .!..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58343 req_id = 9a2dd884-a70d-43eb-8ec7-7c52307b938a
87732025-04-30T06:43:14.860ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2337,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58343 req_id = 9a2dd884-a70d-43eb-8ec7-7c52307b938a
87742025-04-30T06:43:14.860ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
87752025-04-30T06:43:14.860ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58343 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9a2dd884-a70d-43eb-8ec7-7c52307b938a
87762025-04-30T06:43:14.935ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
87772025-04-30T06:43:14.936ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87782025-04-30T06:43:14.936ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87792025-04-30T06:43:14.936ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87802025-04-30T06:43:14.936ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
87812025-04-30T06:43:14.937ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87822025-04-30T06:43:14.937ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ad 8b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53830 req_id = 048ecd86-6396-4447-b936-25ea479c5dea
87832025-04-30T06:43:14.937ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44427,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53830 req_id = 048ecd86-6396-4447-b936-25ea479c5dea
87842025-04-30T06:43:14.938ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
87852025-04-30T06:43:14.941ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44427, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53830 req_id = 048ecd86-6396-4447-b936-25ea479c5dea
87862025-04-30T06:43:14.943ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
87872025-04-30T06:43:14.943ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
87882025-04-30T06:43:14.943ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
87892025-04-30T06:43:14.944ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87902025-04-30T06:43:14.950ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87912025-04-30T06:43:14.950ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87922025-04-30T06:43:14.950ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
87932025-04-30T06:43:14.951ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87942025-04-30T06:43:14.955ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87952025-04-30T06:43:14.955ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
87962025-04-30T06:43:14.956ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87972025-04-30T06:43:14.961ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d27e4306-d6d2-4e3b-b118-ee8c3d2f18da", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
87982025-04-30T06:43:14.978ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
87992025-04-30T06:43:14.981ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f415c88d-c17c-4710-88ed-1d02e0358cf8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88002025-04-30T06:43:14.981ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
88012025-04-30T06:43:15.059ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28a13378-379c-42a9-bc22-ed762b729c0f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88022025-04-30T06:43:15.059ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
88032025-04-30T06:43:15.081ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
88042025-04-30T06:43:15.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdcb5837-25ba-45fd-b057-6e03c53ca832", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88052025-04-30T06:43:15.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
88062025-04-30T06:43:15.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85977399-26b8-411b-8682-a59fb0755617", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88072025-04-30T06:43:15.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
88082025-04-30T06:43:15.091ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8b443dc-1b38-4177-a318-76c9d46d6691", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88092025-04-30T06:43:15.091ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
88102025-04-30T06:43:15.101ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2a42132-fc24-45e9-8dfc-5dc88f88cf7e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88112025-04-30T06:43:15.101ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
88122025-04-30T06:43:15.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d9ce195-435e-4ff2-a329-cbd7bcf232a3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88132025-04-30T06:43:15.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
88142025-04-30T06:43:15.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16db4dec-d5a4-4816-9971-04e2d452289b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88152025-04-30T06:43:15.121ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
88162025-04-30T06:43:15.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ee26b85-e47f-488c-827c-52940625b73f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88172025-04-30T06:43:15.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
88182025-04-30T06:43:15.125ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59d3f986-5586-474b-81ad-bc4410bcc78d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88192025-04-30T06:43:15.125ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
88202025-04-30T06:43:15.126ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c91fca87-7a2d-483a-87a8-9fba5464398f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88212025-04-30T06:43:15.126ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
88222025-04-30T06:43:15.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c0a716a-29c7-496c-8732-3d827b622060", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88232025-04-30T06:43:15.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
88242025-04-30T06:43:15.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97b284fa-a986-4437-85ab-3e52f46cbc12", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88252025-04-30T06:43:15.133ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
88262025-04-30T06:43:15.133ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88272025-04-30T06:43:15.133ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88282025-04-30T06:43:15.133ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88292025-04-30T06:43:15.133ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
88302025-04-30T06:43:15.134ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88312025-04-30T06:43:15.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e8fb5dc-d431-4af0-b818-e422c95a6bd5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88322025-04-30T06:43:15.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
88332025-04-30T06:43:15.137ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88342025-04-30T06:43:15.143ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88352025-04-30T06:43:15.143ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88362025-04-30T06:43:15.144ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
88372025-04-30T06:43:15.144ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88382025-04-30T06:43:15.148ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88392025-04-30T06:43:15.148ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
88402025-04-30T06:43:15.148ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88412025-04-30T06:43:15.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eb16993-370f-4f4d-8d2d-eec3cca9a584", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88422025-04-30T06:43:15.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
88432025-04-30T06:43:15.151ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef648e9a-4382-44b9-a634-213e1b5f6215", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88442025-04-30T06:43:15.152ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
88452025-04-30T06:43:15.154ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00042c22-5856-42e9-87fc-bf6ab152ee02", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88462025-04-30T06:43:15.154ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
88472025-04-30T06:43:15.157ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef714918-34a7-407e-a7a2-d5a30acda90e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88482025-04-30T06:43:15.157ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
88492025-04-30T06:43:15.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12840a4b-c627-4ed6-8560-c73df173c72f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88502025-04-30T06:43:15.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
88512025-04-30T06:43:15.161ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "acfdc767-9bb6-473d-8e68-f143a320df1b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88522025-04-30T06:43:15.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
88532025-04-30T06:43:15.165ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b4b161a-16a9-482d-bb10-9fabc989a5a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88542025-04-30T06:43:15.166ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
88552025-04-30T06:43:15.167ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63cb2a5a-6028-41b7-9ba3-7d8c209406f8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88562025-04-30T06:43:15.167ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
88572025-04-30T06:43:15.180ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec8101ab-6ed2-4aa1-9c34-a3aaa8463b9f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88582025-04-30T06:43:15.180ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
88592025-04-30T06:43:15.184ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
88602025-04-30T06:43:15.185ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b3d2678-dbc7-471b-bd30-b3687668dd25", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88612025-04-30T06:43:15.186ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
88622025-04-30T06:43:15.189ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41d14cff-3198-4d57-bf0f-81f2b051ca09", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88632025-04-30T06:43:15.189ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
88642025-04-30T06:43:15.191ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "024710f0-a7d6-47a4-a40a-b699030be276", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88652025-04-30T06:43:15.191ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
88662025-04-30T06:43:15.193ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6f38aa4-3a69-463e-8cdb-ab3e48438756", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88672025-04-30T06:43:15.194ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
88682025-04-30T06:43:15.197ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bebf97f-35c4-4aa4-9d8b-33092f0e554e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88692025-04-30T06:43:15.197ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
88702025-04-30T06:43:15.201ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "742a482b-c819-4e71-a458-2ffd062773b9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88712025-04-30T06:43:15.202ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
88722025-04-30T06:43:15.206ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d01b760c-d2e5-4560-8e3e-9d31e9b76067", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88732025-04-30T06:43:15.206ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
88742025-04-30T06:43:15.208ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d27f4156-484c-4b87-9695-5321f682226f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88752025-04-30T06:43:15.208ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
88762025-04-30T06:43:15.209ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ae95f4e-62c0-46ff-93b3-11be8e77bb4b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88772025-04-30T06:43:15.210ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
88782025-04-30T06:43:15.214ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cbe8091-4d14-4e26-928e-1a2b8d3fb35f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88792025-04-30T06:43:15.215ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
88802025-04-30T06:43:15.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd8bcc04-5f1e-4b03-8a2c-d140c6132e9d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88812025-04-30T06:43:15.218ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
88822025-04-30T06:43:15.220ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71b53e30-d3ed-4a94-93ef-9181b70b9d07", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88832025-04-30T06:43:15.221ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
88842025-04-30T06:43:15.224ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b04973db-4308-4138-b69d-1f17a5f2b772", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88852025-04-30T06:43:15.224ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
88862025-04-30T06:43:15.227ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76a5df82-40fe-41e1-8211-5cbd385c4416", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88872025-04-30T06:43:15.227ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
88882025-04-30T06:43:15.233ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ec3016f-cf3f-4fae-8ae3-aa6b093e9b7d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88892025-04-30T06:43:15.233ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
88902025-04-30T06:43:15.235ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88912025-04-30T06:43:15.235ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88922025-04-30T06:43:15.235ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88932025-04-30T06:43:15.235ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
88942025-04-30T06:43:15.236ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88952025-04-30T06:43:15.239ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "029ce88e-c815-4631-9f0b-e73ec5c44c46", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
88962025-04-30T06:43:15.239ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
88972025-04-30T06:43:15.239ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88982025-04-30T06:43:15.247ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88992025-04-30T06:43:15.247ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89002025-04-30T06:43:15.247ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
89012025-04-30T06:43:15.247ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89022025-04-30T06:43:15.250ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89032025-04-30T06:43:15.250ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
89042025-04-30T06:43:15.251ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89052025-04-30T06:43:15.252ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "892d1f56-afd7-47cd-b70a-8893c04d4816", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
89062025-04-30T06:43:15.252ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
89072025-04-30T06:43:15.253ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d27ab77f-fb3d-4b35-a40b-91404f9882e2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
89082025-04-30T06:43:15.254ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
89092025-04-30T06:43:15.255ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d12deb93-0c60-471b-af00-69728eaa407a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
89102025-04-30T06:43:15.256ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
89112025-04-30T06:43:15.258ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb3f67b8-fefd-4842-9342-66c54c6210bc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
89122025-04-30T06:43:15.259ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
89132025-04-30T06:43:15.261ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "327917b0-fef1-45df-98b5-a3aab54ba223", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
89142025-04-30T06:43:15.261ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
89152025-04-30T06:43:15.261ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89162025-04-30T06:43:15.261ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89172025-04-30T06:43:15.261ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89182025-04-30T06:43:15.261ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89192025-04-30T06:43:15.267ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89202025-04-30T06:43:15.271ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
89212025-04-30T06:43:15.271ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
89222025-04-30T06:43:15.271ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89232025-04-30T06:43:15.271ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89242025-04-30T06:43:15.271ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89252025-04-30T06:43:15.272ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89262025-04-30T06:43:15.277ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89272025-04-30T06:43:15.286ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
89282025-04-30T06:43:15.287ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89292025-04-30T06:43:15.287ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89302025-04-30T06:43:15.287ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89312025-04-30T06:43:15.363ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
89322025-04-30T06:43:15.363ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
89332025-04-30T06:43:15.364ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:57117 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
89342025-04-30T06:43:15.365ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:57117 req_id = 7ed9b084-c59b-47fe-9ab8-ae2218122c48 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
89352025-04-30T06:43:15.365ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 560 local_addr = [::1]:51119 method = GET remote_addr = [::1]:57117 req_id = 7ed9b084-c59b-47fe-9ab8-ae2218122c48 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
89362025-04-30T06:43:15.366ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed9b084-c59b-47fe-9ab8-ae2218122c48", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
89372025-04-30T06:43:15.371ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
89382025-04-30T06:43:15.371ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
89392025-04-30T06:43:15.371ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
89402025-04-30T06:43:15.373ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
89412025-04-30T06:43:15.373ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89422025-04-30T06:43:15.373ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89432025-04-30T06:43:15.373ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89442025-04-30T06:43:15.374ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89452025-04-30T06:43:15.377ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89462025-04-30T06:43:15.382ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
89472025-04-30T06:43:15.382ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89482025-04-30T06:43:15.382ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89492025-04-30T06:43:15.382ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89502025-04-30T06:43:15.399ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
89512025-04-30T06:43:15.430ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
89522025-04-30T06:43:15.430ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
89532025-04-30T06:43:15.430ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 597.839236ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
89542025-04-30T06:43:15.431ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
89552025-04-30T06:43:15.431ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
89562025-04-30T06:43:15.431ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89572025-04-30T06:43:15.431ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89582025-04-30T06:43:15.431ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89592025-04-30T06:43:15.432ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89602025-04-30T06:43:15.434ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89612025-04-30T06:43:15.439ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
89622025-04-30T06:43:15.439ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89632025-04-30T06:43:15.439ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89642025-04-30T06:43:15.439ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89652025-04-30T06:43:15.443ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
89662025-04-30T06:43:15.443ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
89672025-04-30T06:43:15.445ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d9 7a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .z..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62119 req_id = 7376bd4e-1a50-4144-a97f-66f9376d46c4
89682025-04-30T06:43:15.445ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55674,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62119 req_id = 7376bd4e-1a50-4144-a97f-66f9376d46c4
89692025-04-30T06:43:15.445ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
89702025-04-30T06:43:15.445ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89712025-04-30T06:43:15.445ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62119 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7376bd4e-1a50-4144-a97f-66f9376d46c4
89722025-04-30T06:43:15.446ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4260, tv_nsec: 245125785 } })
89732025-04-30T06:43:15.446ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
89742025-04-30T06:43:15.447ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0c b2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62627 req_id = 327a0ead-388f-491a-9280-cbcafc35793b
89752025-04-30T06:43:15.447ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3250,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62627 req_id = 327a0ead-388f-491a-9280-cbcafc35793b
89762025-04-30T06:43:15.447ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89772025-04-30T06:43:15.447ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62627 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 327a0ead-388f-491a-9280-cbcafc35793b
89782025-04-30T06:43:15.503ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
89792025-04-30T06:43:15.503ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89802025-04-30T06:43:15.503ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89812025-04-30T06:43:15.503ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89822025-04-30T06:43:15.503ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
89832025-04-30T06:43:15.504ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89842025-04-30T06:43:15.505ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c8 b0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51100 req_id = 9b7cec79-94dd-4b95-8fd6-65604c4053d2
89852025-04-30T06:43:15.505ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51376,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51100 req_id = 9b7cec79-94dd-4b95-8fd6-65604c4053d2
89862025-04-30T06:43:15.505ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
89872025-04-30T06:43:15.505ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51100 req_id = 9b7cec79-94dd-4b95-8fd6-65604c4053d2
89882025-04-30T06:43:15.506ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
89892025-04-30T06:43:15.506ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
89902025-04-30T06:43:15.506ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
89912025-04-30T06:43:15.508ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89922025-04-30T06:43:15.515ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89932025-04-30T06:43:15.515ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89942025-04-30T06:43:15.515ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
89952025-04-30T06:43:15.517ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89962025-04-30T06:43:15.519ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "42ffa426-180f-49bb-a5e6-660fbad53b34", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
89972025-04-30T06:43:15.520ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
89982025-04-30T06:43:15.522ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89992025-04-30T06:43:15.522ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
90002025-04-30T06:43:15.523ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90012025-04-30T06:43:15.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d31c1028-548e-4609-96fd-7ee6db539b7e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
90022025-04-30T06:43:15.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
90032025-04-30T06:43:15.529ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42614fe5-26b7-4d8e-a506-2b1e4de8ccd2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
90042025-04-30T06:43:15.531ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
90052025-04-30T06:43:15.538ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0a3241e-c450-48e3-bf6c-956807d00f6c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
90062025-04-30T06:43:15.538ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
90072025-04-30T06:43:15.542ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59071651-2bc9-4dca-8266-f9dee0102c3e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
90082025-04-30T06:43:15.543ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
90092025-04-30T06:43:15.547ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb0e475c-d2aa-43a5-8d6a-eb7e5f87a77c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90102025-04-30T06:43:15.547ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
90112025-04-30T06:43:15.552ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52b2c5c5-bd26-42aa-9dc5-0fb2cbb8e457", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90122025-04-30T06:43:15.552ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
90132025-04-30T06:43:15.563ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
90142025-04-30T06:43:15.563ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3528d7bb-dd86-4b4e-b600-baa5f2a78231", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90152025-04-30T06:43:15.563ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
90162025-04-30T06:43:15.567ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06f287d5-eaaa-4517-95f0-e988129deea2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90172025-04-30T06:43:15.567ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
90182025-04-30T06:43:15.570ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd309ecc-2dab-483b-b876-b6c9aac6b6c3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90192025-04-30T06:43:15.570ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
90202025-04-30T06:43:15.572ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f41145ac-0657-4298-9208-71d3832b887a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90212025-04-30T06:43:15.573ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
90222025-04-30T06:43:15.575ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8b77a7a-9b2a-4d16-aec0-9f2f4acadaa3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90232025-04-30T06:43:15.575ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
90242025-04-30T06:43:15.580ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "334bf10a-7ba9-44c1-8485-a35712ea0c9d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90252025-04-30T06:43:15.580ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
90262025-04-30T06:43:15.583ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
90272025-04-30T06:43:15.584ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90282025-04-30T06:43:15.585ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2fdfe3f1-0072-439a-a2df-88ecef09fa02 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90292025-04-30T06:43:15.585ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2fdfe3f1-0072-439a-a2df-88ecef09fa02 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90302025-04-30T06:43:15.586ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2fdfe3f1-0072-439a-a2df-88ecef09fa02 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90312025-04-30T06:43:15.587ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2fdfe3f1-0072-439a-a2df-88ecef09fa02 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90322025-04-30T06:43:15.592ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 7470 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 2fdfe3f1-0072-439a-a2df-88ecef09fa02 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90332025-04-30T06:43:15.593ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fdfe3f1-0072-439a-a2df-88ecef09fa02", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90342025-04-30T06:43:15.594ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90352025-04-30T06:43:15.595ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = eca419e3-13d0-40aa-8bdb-b860eb834bd6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90362025-04-30T06:43:15.596ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = eca419e3-13d0-40aa-8bdb-b860eb834bd6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90372025-04-30T06:43:15.596ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = eca419e3-13d0-40aa-8bdb-b860eb834bd6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90382025-04-30T06:43:15.598ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = eca419e3-13d0-40aa-8bdb-b860eb834bd6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90392025-04-30T06:43:15.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10942fa8-6e8c-4eb7-9ac1-c50111c32ff5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90402025-04-30T06:43:15.603ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
90412025-04-30T06:43:15.604ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 9208 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = eca419e3-13d0-40aa-8bdb-b860eb834bd6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
90422025-04-30T06:43:15.606ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "eca419e3-13d0-40aa-8bdb-b860eb834bd6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90432025-04-30T06:43:15.607ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
90442025-04-30T06:43:15.608ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90452025-04-30T06:43:15.608ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
90462025-04-30T06:43:15.613ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8fc7459-1a00-4e41-9609-1705a1e77708", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90472025-04-30T06:43:15.614ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
90482025-04-30T06:43:15.614ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90492025-04-30T06:43:15.614ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90502025-04-30T06:43:15.614ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90512025-04-30T06:43:15.614ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
90522025-04-30T06:43:15.616ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90532025-04-30T06:43:15.620ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90542025-04-30T06:43:15.639ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90552025-04-30T06:43:15.639ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90562025-04-30T06:43:15.639ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
90572025-04-30T06:43:15.642ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90582025-04-30T06:43:15.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddb523ae-cf3d-41f4-b8b6-98cb71bfaeec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90592025-04-30T06:43:15.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
90602025-04-30T06:43:15.646ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90612025-04-30T06:43:15.646ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
90622025-04-30T06:43:15.646ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90632025-04-30T06:43:15.649ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88607189-47ba-4f2f-83d3-7a403885001d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90642025-04-30T06:43:15.650ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
90652025-04-30T06:43:15.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9135a47e-e3bc-40c3-a12c-c75fa2a3a548", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90662025-04-30T06:43:15.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
90672025-04-30T06:43:15.668ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da666e9c-fb32-4d07-87b1-a438cfbf2823", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90682025-04-30T06:43:15.668ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
90692025-04-30T06:43:15.670ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7762002c-0aa1-4fe3-b907-26ecc20f22eb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90702025-04-30T06:43:15.670ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
90712025-04-30T06:43:15.673ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac905fd7-cded-4e92-a04b-aeb29aebbcae", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90722025-04-30T06:43:15.673ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
90732025-04-30T06:43:15.675ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4005c5e-6730-4119-81a5-ce1e9b47ef48", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90742025-04-30T06:43:15.675ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
90752025-04-30T06:43:15.676ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f499377b-6fda-4969-aec9-4fe717541056", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90762025-04-30T06:43:15.676ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
90772025-04-30T06:43:15.679ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6399af69-0251-428b-bc0d-ba12f0ba852a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90782025-04-30T06:43:15.680ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
90792025-04-30T06:43:15.682ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90802025-04-30T06:43:15.682ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90812025-04-30T06:43:15.682ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a53a8ad9-3a71-4fe7-96ac-2319ae8ec548", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90822025-04-30T06:43:15.682ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
90832025-04-30T06:43:15.687ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
90842025-04-30T06:43:15.688ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
90852025-04-30T06:43:15.692ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
90862025-04-30T06:43:15.694ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
90872025-04-30T06:43:15.694ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
90882025-04-30T06:43:15.695ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "060141f3-ba47-4d55-888f-e8f842ccaa50", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90892025-04-30T06:43:15.696ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
90902025-04-30T06:43:15.700ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41544664-d8c9-4acb-b057-c5013aaed288", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90912025-04-30T06:43:15.700ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
90922025-04-30T06:43:15.701ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
90932025-04-30T06:43:15.701ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90942025-04-30T06:43:15.704ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cb4c6f9-6027-47dd-907f-adce77924792", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90952025-04-30T06:43:15.704ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
90962025-04-30T06:43:15.713ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49d180da-0df7-4f38-b2f7-a38625b53372", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90972025-04-30T06:43:15.713ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
90982025-04-30T06:43:15.716ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b63c16ff-aa12-48ee-a362-96baa93407dc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
90992025-04-30T06:43:15.717ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
91002025-04-30T06:43:15.719ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a10ad7bf-a60c-458e-8fd8-0eb6f97daa98", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91012025-04-30T06:43:15.719ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
91022025-04-30T06:43:15.719ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9a8656f-5719-4eb0-87b5-8ecef92ca23b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91032025-04-30T06:43:15.720ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
91042025-04-30T06:43:15.722ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c8f8855-59de-48a8-8636-41e4e329bd9b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91052025-04-30T06:43:15.722ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
91062025-04-30T06:43:15.723ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91072025-04-30T06:43:15.723ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91082025-04-30T06:43:15.725ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91092025-04-30T06:43:15.727ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91102025-04-30T06:43:15.728ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91112025-04-30T06:43:15.728ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91122025-04-30T06:43:15.728ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eed4ef77-b21f-46b0-9fc3-41c4c4ec7ef6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91132025-04-30T06:43:15.730ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
91142025-04-30T06:43:15.732ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e968c32-a8b1-4111-9ceb-63ad16fc81b8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91152025-04-30T06:43:15.732ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
91162025-04-30T06:43:15.734ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f7a0c22-54ef-4049-8bfd-2a363faf8f76", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91172025-04-30T06:43:15.734ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
91182025-04-30T06:43:15.737ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "11268ac0-d9e2-448c-9e14-a77807788c98", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91192025-04-30T06:43:15.737ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
91202025-04-30T06:43:15.746ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91212025-04-30T06:43:15.746ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91222025-04-30T06:43:15.746ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91232025-04-30T06:43:15.746ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
91242025-04-30T06:43:15.748ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91252025-04-30T06:43:15.752ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1832a397-e8fd-4068-a1f7-17ae32c2a46b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91262025-04-30T06:43:15.752ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
91272025-04-30T06:43:15.752ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91282025-04-30T06:43:15.758ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91292025-04-30T06:43:15.758ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91302025-04-30T06:43:15.758ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
91312025-04-30T06:43:15.759ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91322025-04-30T06:43:15.762ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91332025-04-30T06:43:15.762ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
91342025-04-30T06:43:15.763ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91352025-04-30T06:43:15.763ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f12ae88b-e861-4abe-8b6d-b834df02f085", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91362025-04-30T06:43:15.764ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
91372025-04-30T06:43:15.771ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ec20156-ec30-47b9-9f88-0458d4004a03", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91382025-04-30T06:43:15.772ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
91392025-04-30T06:43:15.776ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbbc7a71-9ad5-4147-ba7a-0cf72f3db22b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91402025-04-30T06:43:15.776ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
91412025-04-30T06:43:15.778ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab5c73ca-06ed-4f02-be9e-378dd576ec6c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91422025-04-30T06:43:15.778ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
91432025-04-30T06:43:15.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "beae89fb-98b6-4665-a949-fdeb0a961c41", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91442025-04-30T06:43:15.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
91452025-04-30T06:43:15.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c45915e-42d9-4af3-93fe-369956945f78", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91462025-04-30T06:43:15.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
91472025-04-30T06:43:15.782ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "721ddee3-9e7f-4338-8fad-45338ff5fe00", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91482025-04-30T06:43:15.782ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
91492025-04-30T06:43:15.782ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91502025-04-30T06:43:15.782ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91512025-04-30T06:43:15.782ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91522025-04-30T06:43:15.783ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91532025-04-30T06:43:15.783ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
91542025-04-30T06:43:15.786ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91552025-04-30T06:43:15.793ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
91562025-04-30T06:43:15.793ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
91572025-04-30T06:43:15.793ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91582025-04-30T06:43:15.795ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91592025-04-30T06:43:15.795ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91602025-04-30T06:43:15.797ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91612025-04-30T06:43:15.802ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91622025-04-30T06:43:15.809ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
91632025-04-30T06:43:15.809ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91642025-04-30T06:43:15.809ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91652025-04-30T06:43:15.810ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91662025-04-30T06:43:15.880ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
91672025-04-30T06:43:15.880ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
91682025-04-30T06:43:15.882ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:49704 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
91692025-04-30T06:43:15.883ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:49704 req_id = 081217fd-642f-43c6-be17-796fbf1c7fcc sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
91702025-04-30T06:43:15.883ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 555 local_addr = [::1]:51119 method = GET remote_addr = [::1]:49704 req_id = 081217fd-642f-43c6-be17-796fbf1c7fcc response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
91712025-04-30T06:43:15.883ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
91722025-04-30T06:43:15.884ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "081217fd-642f-43c6-be17-796fbf1c7fcc", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
91732025-04-30T06:43:15.885ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
91742025-04-30T06:43:15.886ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
91752025-04-30T06:43:15.886ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
91762025-04-30T06:43:15.888ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
91772025-04-30T06:43:15.888ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91782025-04-30T06:43:15.888ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91792025-04-30T06:43:15.888ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91802025-04-30T06:43:15.889ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91812025-04-30T06:43:15.892ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91822025-04-30T06:43:15.898ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
91832025-04-30T06:43:15.899ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91842025-04-30T06:43:15.899ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91852025-04-30T06:43:15.900ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91862025-04-30T06:43:15.930ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
91872025-04-30T06:43:15.931ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
91882025-04-30T06:43:15.931ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 500.571485ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
91892025-04-30T06:43:15.931ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
91902025-04-30T06:43:15.931ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
91912025-04-30T06:43:15.931ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91922025-04-30T06:43:15.931ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91932025-04-30T06:43:15.931ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91942025-04-30T06:43:15.932ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91952025-04-30T06:43:15.934ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91962025-04-30T06:43:15.935ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91972025-04-30T06:43:15.935ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91982025-04-30T06:43:15.935ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
91992025-04-30T06:43:15.937ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92002025-04-30T06:43:15.938ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92012025-04-30T06:43:15.944ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92022025-04-30T06:43:15.944ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92032025-04-30T06:43:15.944ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92042025-04-30T06:43:15.945ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92052025-04-30T06:43:15.947ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92062025-04-30T06:43:15.956ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92072025-04-30T06:43:15.956ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92082025-04-30T06:43:15.956ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
92092025-04-30T06:43:15.957ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92102025-04-30T06:43:15.960ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92112025-04-30T06:43:15.960ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
92122025-04-30T06:43:15.962ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92132025-04-30T06:43:15.962ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
92142025-04-30T06:43:15.962ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
92152025-04-30T06:43:15.964ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b5 5b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .[..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60137 req_id = 28d424e7-1591-45a8-a6cb-abb4f7047924
92162025-04-30T06:43:15.964ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46427,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60137 req_id = 28d424e7-1591-45a8-a6cb-abb4f7047924
92172025-04-30T06:43:15.964ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
92182025-04-30T06:43:15.964ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
92192025-04-30T06:43:15.964ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60137 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 28d424e7-1591-45a8-a6cb-abb4f7047924
92202025-04-30T06:43:15.965ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4260, tv_nsec: 764126369 } })
92212025-04-30T06:43:15.965ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
92222025-04-30T06:43:15.965ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 17 14 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57529 req_id = 86047acc-b7f7-4df8-84cd-f116e8f3422c
92232025-04-30T06:43:15.966ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5908,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57529 req_id = 86047acc-b7f7-4df8-84cd-f116e8f3422c
92242025-04-30T06:43:15.967ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
92252025-04-30T06:43:15.967ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57529 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 86047acc-b7f7-4df8-84cd-f116e8f3422c
92262025-04-30T06:43:16.048ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
92272025-04-30T06:43:16.050ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 57 fc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c W............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62843 req_id = eee8507e-ed79-46cc-b673-106887870e6c
92282025-04-30T06:43:16.050ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22524,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62843 req_id = eee8507e-ed79-46cc-b673-106887870e6c
92292025-04-30T06:43:16.050ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
92302025-04-30T06:43:16.051ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22524, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62843 req_id = eee8507e-ed79-46cc-b673-106887870e6c
92312025-04-30T06:43:16.052ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
92322025-04-30T06:43:16.052ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
92332025-04-30T06:43:16.052ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
92342025-04-30T06:43:16.074ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "431239d7-88bf-4220-9731-5561708735e6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92352025-04-30T06:43:16.074ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
92362025-04-30T06:43:16.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c6402a4-f8d1-4926-990f-091bc672a91f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92372025-04-30T06:43:16.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
92382025-04-30T06:43:16.119ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bf05226-a660-4062-9d29-ac4a18639b9d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92392025-04-30T06:43:16.119ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
92402025-04-30T06:43:16.124ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
92412025-04-30T06:43:16.126ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50bf61b0-9c0e-444e-bb6b-02008c10fa47", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92422025-04-30T06:43:16.126ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
92432025-04-30T06:43:16.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95def8b1-7095-4735-b86f-0b745b3e115a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92442025-04-30T06:43:16.128ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
92452025-04-30T06:43:16.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2e3ba35-6a5c-4e6b-a92b-fdd2234d56a7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92462025-04-30T06:43:16.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
92472025-04-30T06:43:16.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c44e8e38-6092-4c08-972f-4ff76d1d0808", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92482025-04-30T06:43:16.133ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
92492025-04-30T06:43:16.134ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3a7ad30-27b7-471b-9289-6c8052ee18db", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92502025-04-30T06:43:16.134ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
92512025-04-30T06:43:16.135ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b471a35b-39f8-43a8-ae76-102e40c25593", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92522025-04-30T06:43:16.138ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
92532025-04-30T06:43:16.140ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e913e3d-1304-4f7f-9265-12428e12bb76", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92542025-04-30T06:43:16.140ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
92552025-04-30T06:43:16.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e15cc152-7146-4dea-ba25-3a23e82c0350", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92562025-04-30T06:43:16.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
92572025-04-30T06:43:16.144ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c00f608b-0807-41a9-a4ed-bfcd18fab0ee", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92582025-04-30T06:43:16.145ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
92592025-04-30T06:43:16.148ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "158c04af-4b1b-4f9a-b63a-c37a688603b3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92602025-04-30T06:43:16.150ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
92612025-04-30T06:43:16.150ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05973096-c747-4c24-b9ed-8b43bdb57891", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92622025-04-30T06:43:16.150ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
92632025-04-30T06:43:16.152ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37d96fa2-dc43-445d-bef5-e50298bc5e4f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92642025-04-30T06:43:16.153ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
92652025-04-30T06:43:16.155ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70bc93ae-7f14-4d12-b0ad-c139a5195ccb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92662025-04-30T06:43:16.155ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
92672025-04-30T06:43:16.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "354f2f4d-6f64-4705-8af0-95e181febf5f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92682025-04-30T06:43:16.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
92692025-04-30T06:43:16.159ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d7c7632-9dea-4575-96dc-48b4bc6aa461", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92702025-04-30T06:43:16.159ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
92712025-04-30T06:43:16.161ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ea6ae6-44b4-4ce6-993a-9818c56d9c1d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92722025-04-30T06:43:16.161ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
92732025-04-30T06:43:16.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2de2c436-1cbd-4dd9-a51c-aa0a754b916e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92742025-04-30T06:43:16.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
92752025-04-30T06:43:16.163ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2af1c8eb-8a85-48df-a005-9e4e84d50fd7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92762025-04-30T06:43:16.163ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
92772025-04-30T06:43:16.167ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9d6c689-eed9-4700-8d0b-bf7fa6b43400", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92782025-04-30T06:43:16.167ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
92792025-04-30T06:43:16.168ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f167f5d-888b-469c-8709-a6f57d04d923", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92802025-04-30T06:43:16.168ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
92812025-04-30T06:43:16.171ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33d4412c-239c-4add-b750-e6be86065617", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92822025-04-30T06:43:16.172ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
92832025-04-30T06:43:16.173ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb7e81c8-1eea-45d6-bd19-d5dc2b13654f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92842025-04-30T06:43:16.174ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
92852025-04-30T06:43:16.180ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92862025-04-30T06:43:16.180ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92872025-04-30T06:43:16.180ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92882025-04-30T06:43:16.180ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
92892025-04-30T06:43:16.181ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92902025-04-30T06:43:16.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a199ca90-5ea2-4bd9-8693-9050eb193cae", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92912025-04-30T06:43:16.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
92922025-04-30T06:43:16.183ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92932025-04-30T06:43:16.189ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92942025-04-30T06:43:16.189ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92952025-04-30T06:43:16.189ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
92962025-04-30T06:43:16.190ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92972025-04-30T06:43:16.192ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff8b7b47-ecb5-4f8d-ae68-52e45fe9098d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
92982025-04-30T06:43:16.192ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
92992025-04-30T06:43:16.193ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93002025-04-30T06:43:16.193ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
93012025-04-30T06:43:16.194ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93022025-04-30T06:43:16.195ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a190a962-61ac-4baf-8a60-ab3ad3de74fc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93032025-04-30T06:43:16.196ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
93042025-04-30T06:43:16.198ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eedf1c0-471a-400b-aafe-3f7bc62aa2f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93052025-04-30T06:43:16.198ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
93062025-04-30T06:43:16.201ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6981098d-3ef5-4b26-b91c-220551d4f383", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93072025-04-30T06:43:16.201ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
93082025-04-30T06:43:16.202ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1196b318-6abc-41f2-a578-ba3dabeb833b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93092025-04-30T06:43:16.202ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
93102025-04-30T06:43:16.204ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "777f65c1-a01b-45a0-9250-5802fb8e2dcb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93112025-04-30T06:43:16.204ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
93122025-04-30T06:43:16.206ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a32e6919-6776-43f2-985f-a874562f0a9b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93132025-04-30T06:43:16.206ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
93142025-04-30T06:43:16.207ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ee54063-4f2d-4618-9e0d-71f2954e3939", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93152025-04-30T06:43:16.208ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
93162025-04-30T06:43:16.214ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "566bfdd7-060b-4b12-a2ea-5d746e6ee2b9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93172025-04-30T06:43:16.214ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
93182025-04-30T06:43:16.217ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "300efbb1-ff31-4902-886a-7459169ebb8a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93192025-04-30T06:43:16.217ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
93202025-04-30T06:43:16.231ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6167ff2f-a259-4fbb-aeb7-79f84d8c90b7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93212025-04-30T06:43:16.232ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
93222025-04-30T06:43:16.241ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8243233-4519-4ce3-b886-77b94145ae02", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93232025-04-30T06:43:16.241ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
93242025-04-30T06:43:16.242ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
93252025-04-30T06:43:16.245ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8e02322-79f5-4ae2-a752-e254601d4b3b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93262025-04-30T06:43:16.246ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
93272025-04-30T06:43:16.249ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df6a45e5-8ed1-46bf-9a67-2d34925ee2b6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93282025-04-30T06:43:16.249ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
93292025-04-30T06:43:16.251ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd47fad4-e394-4191-bd65-af96e731d00e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93302025-04-30T06:43:16.251ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
93312025-04-30T06:43:16.252ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db070841-1297-4baf-abb5-d28d42f6505b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93322025-04-30T06:43:16.252ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
93332025-04-30T06:43:16.253ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42b0fb8a-9148-4aef-bbe9-ad898e8efc8d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93342025-04-30T06:43:16.254ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
93352025-04-30T06:43:16.255ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e446fa2-a58f-4127-b24e-5757c70ab461", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93362025-04-30T06:43:16.255ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
93372025-04-30T06:43:16.256ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76cbd4d8-4582-48c9-b89b-0fe44a16a9e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93382025-04-30T06:43:16.256ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
93392025-04-30T06:43:16.256ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93402025-04-30T06:43:16.256ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93412025-04-30T06:43:16.256ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93422025-04-30T06:43:16.257ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93432025-04-30T06:43:16.260ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93442025-04-30T06:43:16.274ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
93452025-04-30T06:43:16.274ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
93462025-04-30T06:43:16.274ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93472025-04-30T06:43:16.274ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93482025-04-30T06:43:16.274ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93492025-04-30T06:43:16.274ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93502025-04-30T06:43:16.279ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93512025-04-30T06:43:16.284ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
93522025-04-30T06:43:16.285ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93532025-04-30T06:43:16.285ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93542025-04-30T06:43:16.285ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93552025-04-30T06:43:16.355ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
93562025-04-30T06:43:16.355ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
93572025-04-30T06:43:16.357ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93582025-04-30T06:43:16.357ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93592025-04-30T06:43:16.357ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93602025-04-30T06:43:16.357ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
93612025-04-30T06:43:16.359ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93622025-04-30T06:43:16.360ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:43591 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
93632025-04-30T06:43:16.361ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:43591 req_id = 75b8aefa-b775-41b3-95a6-c497daa9af16 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
93642025-04-30T06:43:16.362ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 949 local_addr = [::1]:51119 method = GET remote_addr = [::1]:43591 req_id = 75b8aefa-b775-41b3-95a6-c497daa9af16 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
93652025-04-30T06:43:16.362ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "75b8aefa-b775-41b3-95a6-c497daa9af16", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
93662025-04-30T06:43:16.365ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
93672025-04-30T06:43:16.374ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
93682025-04-30T06:43:16.374ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
93692025-04-30T06:43:16.387ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
93702025-04-30T06:43:16.388ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93712025-04-30T06:43:16.388ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93722025-04-30T06:43:16.388ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93732025-04-30T06:43:16.389ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93742025-04-30T06:43:16.389ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93752025-04-30T06:43:16.402ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93762025-04-30T06:43:16.402ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93772025-04-30T06:43:16.403ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
93782025-04-30T06:43:16.403ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93792025-04-30T06:43:16.406ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93802025-04-30T06:43:16.412ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93812025-04-30T06:43:16.413ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93822025-04-30T06:43:16.413ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93832025-04-30T06:43:16.414ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93842025-04-30T06:43:16.416ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93852025-04-30T06:43:16.416ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
93862025-04-30T06:43:16.416ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93872025-04-30T06:43:16.449ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
93882025-04-30T06:43:16.449ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
93892025-04-30T06:43:16.449ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
93902025-04-30T06:43:16.449ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 518.200591ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
93912025-04-30T06:43:16.449ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
93922025-04-30T06:43:16.449ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
93932025-04-30T06:43:16.450ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93942025-04-30T06:43:16.450ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93952025-04-30T06:43:16.450ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93962025-04-30T06:43:16.450ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93972025-04-30T06:43:16.453ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93982025-04-30T06:43:16.460ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93992025-04-30T06:43:16.460ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94002025-04-30T06:43:16.460ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94012025-04-30T06:43:16.460ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94022025-04-30T06:43:16.465ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
94032025-04-30T06:43:16.465ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
94042025-04-30T06:43:16.473ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3a 50 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 :P..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65400 req_id = 87526a7e-1e7e-454c-af5c-0d736076c114
94052025-04-30T06:43:16.473ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14928,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65400 req_id = 87526a7e-1e7e-454c-af5c-0d736076c114
94062025-04-30T06:43:16.473ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
94072025-04-30T06:43:16.474ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
94082025-04-30T06:43:16.474ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65400 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 87526a7e-1e7e-454c-af5c-0d736076c114
94092025-04-30T06:43:16.476ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4261, tv_nsec: 275009159 } })
94102025-04-30T06:43:16.476ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
94112025-04-30T06:43:16.476ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 78 48 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e xH..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57415 req_id = 255ecdf8-6182-4ba8-a19c-ad71c0a88067
94122025-04-30T06:43:16.476ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30792,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57415 req_id = 255ecdf8-6182-4ba8-a19c-ad71c0a88067
94132025-04-30T06:43:16.476ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
94142025-04-30T06:43:16.477ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57415 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 255ecdf8-6182-4ba8-a19c-ad71c0a88067
94152025-04-30T06:43:16.539ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
94162025-04-30T06:43:16.540ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94172025-04-30T06:43:16.540ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94182025-04-30T06:43:16.540ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94192025-04-30T06:43:16.540ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94202025-04-30T06:43:16.541ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94212025-04-30T06:43:16.542ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f5 82 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52005 req_id = 68937e04-710f-4604-8e93-502d3a05fb9c
94222025-04-30T06:43:16.542ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62850,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52005 req_id = 68937e04-710f-4604-8e93-502d3a05fb9c
94232025-04-30T06:43:16.542ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
94242025-04-30T06:43:16.542ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62850, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52005 req_id = 68937e04-710f-4604-8e93-502d3a05fb9c
94252025-04-30T06:43:16.544ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
94262025-04-30T06:43:16.545ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
94272025-04-30T06:43:16.545ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
94282025-04-30T06:43:16.548ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94292025-04-30T06:43:16.561ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94302025-04-30T06:43:16.561ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94312025-04-30T06:43:16.561ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94322025-04-30T06:43:16.563ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94332025-04-30T06:43:16.567ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccf67618-10a6-4c55-b7d7-5b6b304a3d55", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94342025-04-30T06:43:16.567ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
94352025-04-30T06:43:16.568ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94362025-04-30T06:43:16.568ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94372025-04-30T06:43:16.569ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94382025-04-30T06:43:16.570ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31254229-cdc1-4006-959c-7f23ec396449", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94392025-04-30T06:43:16.570ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
94402025-04-30T06:43:16.584ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a4ac4f7-1418-43a6-9646-a8fc6de5029a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94412025-04-30T06:43:16.584ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
94422025-04-30T06:43:16.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62865645-0735-4733-9b99-1d4a1980e153", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94432025-04-30T06:43:16.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
94442025-04-30T06:43:16.610ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0934da68-def3-47eb-8a8e-31aaeceacf79", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94452025-04-30T06:43:16.611ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
94462025-04-30T06:43:16.615ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bc467de-df2a-4888-8d23-7d36655a1803", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94472025-04-30T06:43:16.615ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
94482025-04-30T06:43:16.618ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9c9098e-f10a-4918-95f6-0606e479c467", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94492025-04-30T06:43:16.619ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
94502025-04-30T06:43:16.622ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
94512025-04-30T06:43:16.624ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7daf1f9e-fd5b-442c-a803-278a89cd4463", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94522025-04-30T06:43:16.624ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
94532025-04-30T06:43:16.625ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "641fe761-decb-485d-8b39-fefa532c0834", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94542025-04-30T06:43:16.625ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
94552025-04-30T06:43:16.627ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2c9312d-be2d-4f05-9856-aff3aaa79a0f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94562025-04-30T06:43:16.628ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
94572025-04-30T06:43:16.629ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cefcc89f-ec72-428b-9d80-2b547947f401", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94582025-04-30T06:43:16.629ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
94592025-04-30T06:43:16.633ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6031958d-d69c-4abe-844b-0fd57a9b0b62", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94602025-04-30T06:43:16.633ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
94612025-04-30T06:43:16.634ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49f4dbec-ef0d-48ce-b1b9-6bfca5e12c6e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94622025-04-30T06:43:16.635ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
94632025-04-30T06:43:16.637ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "070b30c9-fb2a-4128-a524-c506c0b5b902", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94642025-04-30T06:43:16.637ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
94652025-04-30T06:43:16.640ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8669c565-70a4-4b0e-9974-a695944a2cd1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94662025-04-30T06:43:16.640ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
94672025-04-30T06:43:16.641ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58919e4a-7a7d-4139-8c17-ed228cbfe604", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94682025-04-30T06:43:16.641ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
94692025-04-30T06:43:16.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6387ed1-89a0-4b95-a645-d7b6773ff087", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94702025-04-30T06:43:16.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
94712025-04-30T06:43:16.647ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e217be72-bdc1-4ae3-abcc-3af2e86f71ca", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94722025-04-30T06:43:16.647ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
94732025-04-30T06:43:16.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4607084b-439e-4c98-8b62-25870a4b6096", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94742025-04-30T06:43:16.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
94752025-04-30T06:43:16.649ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c93266c4-7929-4740-a61a-432e09f3d20a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94762025-04-30T06:43:16.650ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
94772025-04-30T06:43:16.650ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3700719-9a28-453a-b762-2cafd893cd03", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94782025-04-30T06:43:16.651ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
94792025-04-30T06:43:16.654ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e164d0ea-33a4-4bf3-a19b-975efbdd3f09", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94802025-04-30T06:43:16.654ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
94812025-04-30T06:43:16.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "909e7ade-6973-4ff6-b1b2-80f73094d9fa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94822025-04-30T06:43:16.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
94832025-04-30T06:43:16.656ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94f9d77d-93f5-4b3b-acd8-cf261abac5dc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94842025-04-30T06:43:16.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
94852025-04-30T06:43:16.659ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fb07370-e012-4d0b-9c14-aaf33647a064", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94862025-04-30T06:43:16.659ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
94872025-04-30T06:43:16.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99079463-e727-48eb-8ab8-5cd316cd6563", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94882025-04-30T06:43:16.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
94892025-04-30T06:43:16.667ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cf0798f-396f-4039-833f-d4abefcd9451", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94902025-04-30T06:43:16.668ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
94912025-04-30T06:43:16.678ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5c3feda-a4d4-434a-aac3-02e2f2ac4212", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
94922025-04-30T06:43:16.679ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
94932025-04-30T06:43:16.679ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94942025-04-30T06:43:16.679ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94952025-04-30T06:43:16.679ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94962025-04-30T06:43:16.679ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94972025-04-30T06:43:16.681ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94982025-04-30T06:43:16.682ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94992025-04-30T06:43:16.682ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95002025-04-30T06:43:16.682ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
95012025-04-30T06:43:16.684ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
95022025-04-30T06:43:16.689ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95032025-04-30T06:43:16.695ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95042025-04-30T06:43:16.695ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95052025-04-30T06:43:16.695ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
95062025-04-30T06:43:16.696ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95072025-04-30T06:43:16.696ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
95082025-04-30T06:43:16.697ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
95092025-04-30T06:43:16.701ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
95102025-04-30T06:43:16.702ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95112025-04-30T06:43:16.703ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95122025-04-30T06:43:16.703ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
95132025-04-30T06:43:16.703ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95142025-04-30T06:43:16.708ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a202a5c3-0325-497f-ae8d-277b07471f9c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95152025-04-30T06:43:16.708ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
95162025-04-30T06:43:16.712ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "298542c4-6e43-43a1-9919-c7e1c4b3f3cd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95172025-04-30T06:43:16.713ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
95182025-04-30T06:43:16.714ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a6db3c2-ef37-4d10-95e8-63878fb55881", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95192025-04-30T06:43:16.714ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
95202025-04-30T06:43:16.715ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b528f04a-bab6-45a9-b490-65a8809358b2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95212025-04-30T06:43:16.715ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
95222025-04-30T06:43:16.716ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f718fdf3-cbba-4deb-9f11-11b8cdf81ad8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95232025-04-30T06:43:16.716ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
95242025-04-30T06:43:16.718ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "242cf5f4-f790-4442-970a-9e5c2b2ed272", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95252025-04-30T06:43:16.718ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
95262025-04-30T06:43:16.720ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6394e24f-ac90-479e-8d7e-08af8f6dfdce", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95272025-04-30T06:43:16.720ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
95282025-04-30T06:43:16.722ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98b74c80-17c8-4aec-a240-2066efd61031", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95292025-04-30T06:43:16.722ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
95302025-04-30T06:43:16.723ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95312025-04-30T06:43:16.723ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95322025-04-30T06:43:16.726ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95332025-04-30T06:43:16.728ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95342025-04-30T06:43:16.728ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95352025-04-30T06:43:16.728ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95362025-04-30T06:43:16.728ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08318d52-fa3d-4f5a-836a-d0e963971cde", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95372025-04-30T06:43:16.728ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
95382025-04-30T06:43:16.730ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0685bdef-e42a-4081-9ad3-4b5801bc59bf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95392025-04-30T06:43:16.730ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
95402025-04-30T06:43:16.735ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92aea156-5873-4a8c-9b5e-c52189af0039", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95412025-04-30T06:43:16.735ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
95422025-04-30T06:43:16.737ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
95432025-04-30T06:43:16.738ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "226260bc-d950-4716-baa4-695c861914f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95442025-04-30T06:43:16.738ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
95452025-04-30T06:43:16.741ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cf3a236-f21c-4109-b765-242f14916721", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95462025-04-30T06:43:16.742ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
95472025-04-30T06:43:16.746ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8004cbf2-756d-449c-b953-689105df85f3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95482025-04-30T06:43:16.746ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
95492025-04-30T06:43:16.747ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9b31382-3ef3-4b7a-8a90-da2835556667", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95502025-04-30T06:43:16.747ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
95512025-04-30T06:43:16.749ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12973616-7007-441a-8472-80a0deb6e5e9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95522025-04-30T06:43:16.749ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
95532025-04-30T06:43:16.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddee7b4c-964f-4aee-9051-07f6c6a2d034", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95542025-04-30T06:43:16.750ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
95552025-04-30T06:43:16.750ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95562025-04-30T06:43:16.750ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95572025-04-30T06:43:16.750ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95582025-04-30T06:43:16.752ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95592025-04-30T06:43:16.755ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95602025-04-30T06:43:16.762ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
95612025-04-30T06:43:16.762ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
95622025-04-30T06:43:16.762ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95632025-04-30T06:43:16.762ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95642025-04-30T06:43:16.762ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95652025-04-30T06:43:16.763ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95662025-04-30T06:43:16.766ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95672025-04-30T06:43:16.780ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
95682025-04-30T06:43:16.780ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95692025-04-30T06:43:16.780ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95702025-04-30T06:43:16.781ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95712025-04-30T06:43:16.784ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
95722025-04-30T06:43:16.858ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
95732025-04-30T06:43:16.859ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
95742025-04-30T06:43:16.859ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95752025-04-30T06:43:16.859ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95762025-04-30T06:43:16.860ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95772025-04-30T06:43:16.860ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
95782025-04-30T06:43:16.861ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95792025-04-30T06:43:16.861ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:57764 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
95802025-04-30T06:43:16.861ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:57764 req_id = c8a6890d-3415-4141-8a57-3c00b9227e46 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
95812025-04-30T06:43:16.862ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 678 local_addr = [::1]:51119 method = GET remote_addr = [::1]:57764 req_id = c8a6890d-3415-4141-8a57-3c00b9227e46 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
95822025-04-30T06:43:16.863ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8a6890d-3415-4141-8a57-3c00b9227e46", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
95832025-04-30T06:43:16.868ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95842025-04-30T06:43:16.868ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95852025-04-30T06:43:16.868ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
95862025-04-30T06:43:16.871ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95872025-04-30T06:43:16.871ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95882025-04-30T06:43:16.871ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95892025-04-30T06:43:16.871ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95902025-04-30T06:43:16.872ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95912025-04-30T06:43:16.874ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95922025-04-30T06:43:16.881ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95932025-04-30T06:43:16.881ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95942025-04-30T06:43:16.881ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
95952025-04-30T06:43:16.881ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95962025-04-30T06:43:16.883ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95972025-04-30T06:43:16.890ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95982025-04-30T06:43:16.890ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95992025-04-30T06:43:16.890ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96002025-04-30T06:43:16.891ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96012025-04-30T06:43:16.893ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96022025-04-30T06:43:16.893ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96032025-04-30T06:43:16.894ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96042025-04-30T06:43:16.930ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
96052025-04-30T06:43:16.934ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
96062025-04-30T06:43:16.934ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 484.634502ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
96072025-04-30T06:43:16.934ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
96082025-04-30T06:43:16.934ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
96092025-04-30T06:43:16.934ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96102025-04-30T06:43:16.934ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96112025-04-30T06:43:16.934ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96122025-04-30T06:43:16.935ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96132025-04-30T06:43:16.939ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96142025-04-30T06:43:16.946ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
96152025-04-30T06:43:16.946ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96162025-04-30T06:43:16.946ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96172025-04-30T06:43:16.947ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96182025-04-30T06:43:16.952ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
96192025-04-30T06:43:16.954ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
96202025-04-30T06:43:16.955ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
96212025-04-30T06:43:16.955ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 02 73 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .s..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63420 req_id = 6e53dbd5-f1de-4f19-9647-37584420b360
96222025-04-30T06:43:16.958ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 627,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63420 req_id = 6e53dbd5-f1de-4f19-9647-37584420b360
96232025-04-30T06:43:16.959ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
96242025-04-30T06:43:16.959ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
96252025-04-30T06:43:16.959ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63420 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6e53dbd5-f1de-4f19-9647-37584420b360
96262025-04-30T06:43:16.959ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4261, tv_nsec: 758719731 } })
96272025-04-30T06:43:16.959ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
96282025-04-30T06:43:16.960ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ae 39 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .9..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56081 req_id = 22303691-a74b-45b1-b947-508dd03947c8
96292025-04-30T06:43:16.960ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44601,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56081 req_id = 22303691-a74b-45b1-b947-508dd03947c8
96302025-04-30T06:43:16.960ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
96312025-04-30T06:43:16.960ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56081 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 22303691-a74b-45b1-b947-508dd03947c8
96322025-04-30T06:43:17.032ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
96332025-04-30T06:43:17.032ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96342025-04-30T06:43:17.032ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96352025-04-30T06:43:17.032ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96362025-04-30T06:43:17.032ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96372025-04-30T06:43:17.037ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96382025-04-30T06:43:17.040ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f2 0f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58396 req_id = 4b7b0b97-b3cb-4735-bc14-def3ea53c405
96392025-04-30T06:43:17.040ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61967,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58396 req_id = 4b7b0b97-b3cb-4735-bc14-def3ea53c405
96402025-04-30T06:43:17.040ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
96412025-04-30T06:43:17.040ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61967, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58396 req_id = 4b7b0b97-b3cb-4735-bc14-def3ea53c405
96422025-04-30T06:43:17.041ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
96432025-04-30T06:43:17.041ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
96442025-04-30T06:43:17.041ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
96452025-04-30T06:43:17.043ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96462025-04-30T06:43:17.060ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96472025-04-30T06:43:17.060ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96482025-04-30T06:43:17.060ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96492025-04-30T06:43:17.062ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96502025-04-30T06:43:17.079ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96512025-04-30T06:43:17.079ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96522025-04-30T06:43:17.079ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96532025-04-30T06:43:17.095ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0aea813c-e52f-48f5-ae2c-a2f27d1363e3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96542025-04-30T06:43:17.096ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
96552025-04-30T06:43:17.117ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3c5f09b-f837-4ca9-8ccd-bd1458955881", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96562025-04-30T06:43:17.117ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
96572025-04-30T06:43:17.126ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2aa75ae-90e4-4190-85a3-6f09ddb37c61", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96582025-04-30T06:43:17.127ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
96592025-04-30T06:43:17.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d80072c3-401c-4f9c-b165-eaccf2f0f0db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96602025-04-30T06:43:17.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
96612025-04-30T06:43:17.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c302a75-9118-4083-82e9-952a18802aa3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96622025-04-30T06:43:17.133ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
96632025-04-30T06:43:17.135ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c91e7d44-5693-4a78-9c15-5cff70237402", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96642025-04-30T06:43:17.135ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
96652025-04-30T06:43:17.137ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6847a2aa-da8a-442b-9592-4c1530514705", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96662025-04-30T06:43:17.137ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
96672025-04-30T06:43:17.140ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4679bdb-7671-42ee-a705-9dc222ee8552", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96682025-04-30T06:43:17.140ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
96692025-04-30T06:43:17.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67740562-e71e-4dff-bf0e-8efb903e18bc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96702025-04-30T06:43:17.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
96712025-04-30T06:43:17.143ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
96722025-04-30T06:43:17.151ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5412869-5cf5-4474-92e4-d416007fc403", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96732025-04-30T06:43:17.151ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
96742025-04-30T06:43:17.152ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba5ade12-6b96-4a36-884a-13b6f0039912", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96752025-04-30T06:43:17.180ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
96762025-04-30T06:43:17.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7406a418-a16b-4fd7-9f2f-488b842c675a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96772025-04-30T06:43:17.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
96782025-04-30T06:43:17.184ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c613c57-532e-47c5-aaea-2e3d92cc4731", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96792025-04-30T06:43:17.184ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
96802025-04-30T06:43:17.187ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0ef74b3-36c1-40af-a40f-95a292c0f5d2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96812025-04-30T06:43:17.188ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
96822025-04-30T06:43:17.191ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f850740-23f8-457a-a075-dea9e2bdb768", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96832025-04-30T06:43:17.192ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
96842025-04-30T06:43:17.195ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d45208fa-d5b7-47fe-b05a-72b44204ff0f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96852025-04-30T06:43:17.195ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
96862025-04-30T06:43:17.195ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96872025-04-30T06:43:17.195ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96882025-04-30T06:43:17.195ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96892025-04-30T06:43:17.195ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96902025-04-30T06:43:17.200ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96912025-04-30T06:43:17.201ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e35ccfc-2fdf-4956-935e-a3d2ee3953af", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96922025-04-30T06:43:17.201ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
96932025-04-30T06:43:17.201ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96942025-04-30T06:43:17.209ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96952025-04-30T06:43:17.209ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96962025-04-30T06:43:17.209ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96972025-04-30T06:43:17.210ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96982025-04-30T06:43:17.213ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9c1cf96-d4bd-4274-ad53-d2b2811cb406", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
96992025-04-30T06:43:17.213ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
97002025-04-30T06:43:17.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6972e4ee-75e4-4577-b415-acc96bb4b9f2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97012025-04-30T06:43:17.216ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
97022025-04-30T06:43:17.217ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97032025-04-30T06:43:17.217ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
97042025-04-30T06:43:17.218ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97052025-04-30T06:43:17.219ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2191507a-c7dc-45c7-b5c2-2bf0d7379789", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97062025-04-30T06:43:17.220ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
97072025-04-30T06:43:17.221ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c62f32a-5ac5-4bf3-8652-86a7c8de7a96", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97082025-04-30T06:43:17.221ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
97092025-04-30T06:43:17.224ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37ea0776-76f4-42fe-adc0-3f6a9c180527", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97102025-04-30T06:43:17.224ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
97112025-04-30T06:43:17.224ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db4fffbe-3b78-4c7f-b3a6-6d01f0cc1f07", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97122025-04-30T06:43:17.224ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
97132025-04-30T06:43:17.226ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc662ab6-5baa-499f-b333-d44c5013c28e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97142025-04-30T06:43:17.226ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
97152025-04-30T06:43:17.229ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98b9f518-d919-48eb-bfb7-205d48ae83b3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97162025-04-30T06:43:17.229ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
97172025-04-30T06:43:17.232ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d736e22a-826a-43f8-9a8a-e045c00e857f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97182025-04-30T06:43:17.233ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
97192025-04-30T06:43:17.235ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b93b1d8-7bea-4c88-a228-73f72e241468", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97202025-04-30T06:43:17.235ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
97212025-04-30T06:43:17.238ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7899283-5f1d-4cbe-be36-9e4ea1f77d71", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97222025-04-30T06:43:17.240ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
97232025-04-30T06:43:17.248ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b998e954-6e64-401e-af6e-576b037e3188", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97242025-04-30T06:43:17.248ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
97252025-04-30T06:43:17.251ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "daa0bd45-5b0d-4033-8e61-149ffaa0449e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97262025-04-30T06:43:17.252ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
97272025-04-30T06:43:17.253ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78c910bd-2771-46ae-9fd7-91ca981e1dd3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97282025-04-30T06:43:17.253ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
97292025-04-30T06:43:17.254ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2fa1225-696f-4b95-95cc-8842eb2c0fb6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97302025-04-30T06:43:17.254ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
97312025-04-30T06:43:17.256ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03df85a8-8f77-41b7-a18d-39d54eb6a25b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97322025-04-30T06:43:17.256ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
97332025-04-30T06:43:17.257ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b339fb8-311e-429a-b6e4-8ffb2d81f4a1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97342025-04-30T06:43:17.257ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
97352025-04-30T06:43:17.259ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "292762a4-f759-43dd-8023-558aa4f5a28a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97362025-04-30T06:43:17.259ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
97372025-04-30T06:43:17.262ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df27ecc9-e2b8-4c0b-91ea-98811561109a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97382025-04-30T06:43:17.263ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
97392025-04-30T06:43:17.266ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "999e65be-61b9-4a0c-9b7e-4e2dc8334364", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97402025-04-30T06:43:17.266ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
97412025-04-30T06:43:17.268ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d75c13f-7042-456a-9a86-953ae80d6746", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97422025-04-30T06:43:17.268ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
97432025-04-30T06:43:17.273ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bac4f9e-54e6-42e0-be5b-aefef8062962", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97442025-04-30T06:43:17.273ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
97452025-04-30T06:43:17.274ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
97462025-04-30T06:43:17.276ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb7866bc-6221-4fa7-83f1-3c96120279de", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97472025-04-30T06:43:17.276ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
97482025-04-30T06:43:17.278ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c1c0d2a-5076-47ea-8c90-ecb206d49523", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97492025-04-30T06:43:17.279ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
97502025-04-30T06:43:17.280ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5d2fbbc-c3b9-499b-bb23-53cc7918a9cb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97512025-04-30T06:43:17.280ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
97522025-04-30T06:43:17.281ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc08f0ce-e322-4c31-8bc3-91ed978f9c93", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97532025-04-30T06:43:17.282ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
97542025-04-30T06:43:17.283ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb5126fb-a2a0-4ad1-a19e-1817fc13ec9b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97552025-04-30T06:43:17.284ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
97562025-04-30T06:43:17.285ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fd86d92-ef75-4ade-afc5-d6c54106200c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97572025-04-30T06:43:17.285ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
97582025-04-30T06:43:17.285ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97592025-04-30T06:43:17.285ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97602025-04-30T06:43:17.285ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97612025-04-30T06:43:17.286ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97622025-04-30T06:43:17.292ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97632025-04-30T06:43:17.297ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97642025-04-30T06:43:17.297ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
97652025-04-30T06:43:17.297ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97662025-04-30T06:43:17.297ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97672025-04-30T06:43:17.297ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97682025-04-30T06:43:17.297ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97692025-04-30T06:43:17.299ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97702025-04-30T06:43:17.304ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97712025-04-30T06:43:17.304ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97722025-04-30T06:43:17.304ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97732025-04-30T06:43:17.305ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97742025-04-30T06:43:17.375ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
97752025-04-30T06:43:17.376ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
97762025-04-30T06:43:17.376ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97772025-04-30T06:43:17.377ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97782025-04-30T06:43:17.377ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97792025-04-30T06:43:17.377ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
97802025-04-30T06:43:17.377ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97812025-04-30T06:43:17.378ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:48971 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
97822025-04-30T06:43:17.380ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:48971 req_id = a9116064-89ab-42c2-a440-cb7f61bc5185 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97832025-04-30T06:43:17.380ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 659 local_addr = [::1]:51119 method = GET remote_addr = [::1]:48971 req_id = a9116064-89ab-42c2-a440-cb7f61bc5185 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97842025-04-30T06:43:17.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9116064-89ab-42c2-a440-cb7f61bc5185", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
97852025-04-30T06:43:17.396ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97862025-04-30T06:43:17.396ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97872025-04-30T06:43:17.396ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
97882025-04-30T06:43:17.399ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97892025-04-30T06:43:17.399ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97902025-04-30T06:43:17.399ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97912025-04-30T06:43:17.399ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97922025-04-30T06:43:17.400ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97932025-04-30T06:43:17.401ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97942025-04-30T06:43:17.411ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97952025-04-30T06:43:17.412ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97962025-04-30T06:43:17.415ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
97972025-04-30T06:43:17.417ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97982025-04-30T06:43:17.421ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97992025-04-30T06:43:17.421ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
98002025-04-30T06:43:17.422ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98012025-04-30T06:43:17.430ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98022025-04-30T06:43:17.440ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
98032025-04-30T06:43:17.441ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98042025-04-30T06:43:17.441ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98052025-04-30T06:43:17.442ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98062025-04-30T06:43:17.484ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
98072025-04-30T06:43:17.485ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
98082025-04-30T06:43:17.486ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
98092025-04-30T06:43:17.486ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 552.126656ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
98102025-04-30T06:43:17.486ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
98112025-04-30T06:43:17.486ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
98122025-04-30T06:43:17.487ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98132025-04-30T06:43:17.487ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98142025-04-30T06:43:17.487ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98152025-04-30T06:43:17.488ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98162025-04-30T06:43:17.491ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98172025-04-30T06:43:17.497ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
98182025-04-30T06:43:17.497ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98192025-04-30T06:43:17.497ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98202025-04-30T06:43:17.498ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98212025-04-30T06:43:17.501ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
98222025-04-30T06:43:17.501ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
98232025-04-30T06:43:17.502ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bb 37 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .7..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59677 req_id = 909c850a-61bf-439c-a205-55f0635585b1
98242025-04-30T06:43:17.502ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47927,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59677 req_id = 909c850a-61bf-439c-a205-55f0635585b1
98252025-04-30T06:43:17.502ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
98262025-04-30T06:43:17.503ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
98272025-04-30T06:43:17.503ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59677 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 909c850a-61bf-439c-a205-55f0635585b1
98282025-04-30T06:43:17.503ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4262, tv_nsec: 302697611 } })
98292025-04-30T06:43:17.503ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
98302025-04-30T06:43:17.504ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5b 8a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e [...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65199 req_id = aa22c2c4-6fc5-41e1-9af1-7e216741880f
98312025-04-30T06:43:17.504ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23434,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65199 req_id = aa22c2c4-6fc5-41e1-9af1-7e216741880f
98322025-04-30T06:43:17.505ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
98332025-04-30T06:43:17.505ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65199 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = aa22c2c4-6fc5-41e1-9af1-7e216741880f
98342025-04-30T06:43:17.553ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
98352025-04-30T06:43:17.554ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98362025-04-30T06:43:17.554ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98372025-04-30T06:43:17.554ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98382025-04-30T06:43:17.554ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
98392025-04-30T06:43:17.555ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98402025-04-30T06:43:17.555ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1b 82 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63342 req_id = 16a91db4-b11a-428c-bac7-73f7f2f3110d
98412025-04-30T06:43:17.555ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7042,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63342 req_id = 16a91db4-b11a-428c-bac7-73f7f2f3110d
98422025-04-30T06:43:17.556ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
98432025-04-30T06:43:17.556ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7042, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63342 req_id = 16a91db4-b11a-428c-bac7-73f7f2f3110d
98442025-04-30T06:43:17.556ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
98452025-04-30T06:43:17.556ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
98462025-04-30T06:43:17.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
98472025-04-30T06:43:17.558ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98482025-04-30T06:43:17.567ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98492025-04-30T06:43:17.567ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98502025-04-30T06:43:17.568ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
98512025-04-30T06:43:17.568ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98522025-04-30T06:43:17.576ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98532025-04-30T06:43:17.576ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
98542025-04-30T06:43:17.577ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98552025-04-30T06:43:17.578ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dce5d339-eb18-4ed5-80bf-6c07ea360908", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98562025-04-30T06:43:17.578ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
98572025-04-30T06:43:17.582ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
98582025-04-30T06:43:17.583ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
98592025-04-30T06:43:17.583ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 24c50479-26dc-4670-8290-69073168d78c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
98602025-04-30T06:43:17.584ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 24c50479-26dc-4670-8290-69073168d78c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
98612025-04-30T06:43:17.584ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 24c50479-26dc-4670-8290-69073168d78c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
98622025-04-30T06:43:17.586ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 24c50479-26dc-4670-8290-69073168d78c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
98632025-04-30T06:43:17.586ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16310519-c571-467a-b067-bca717b5ce47", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98642025-04-30T06:43:17.586ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
98652025-04-30T06:43:17.590ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 7092 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 24c50479-26dc-4670-8290-69073168d78c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
98662025-04-30T06:43:17.591ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "24c50479-26dc-4670-8290-69073168d78c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98672025-04-30T06:43:17.591ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
98682025-04-30T06:43:17.593ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 461e9437-9dea-4be2-b859-aa430580b90b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
98692025-04-30T06:43:17.593ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 461e9437-9dea-4be2-b859-aa430580b90b resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
98702025-04-30T06:43:17.593ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 461e9437-9dea-4be2-b859-aa430580b90b roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
98712025-04-30T06:43:17.594ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 461e9437-9dea-4be2-b859-aa430580b90b resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
98722025-04-30T06:43:17.601ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 8855 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 461e9437-9dea-4be2-b859-aa430580b90b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
98732025-04-30T06:43:17.602ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "461e9437-9dea-4be2-b859-aa430580b90b", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98742025-04-30T06:43:17.602ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
98752025-04-30T06:43:17.602ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
98762025-04-30T06:43:17.602ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
98772025-04-30T06:43:17.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dadb3386-b407-43a7-9abd-82e78bff7ac8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98782025-04-30T06:43:17.603ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
98792025-04-30T06:43:17.625ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a516730a-61fd-480b-b417-7338001d9989", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98802025-04-30T06:43:17.625ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
98812025-04-30T06:43:17.631ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93b18ce1-8356-4528-9bcd-0f32a9225e2a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98822025-04-30T06:43:17.631ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
98832025-04-30T06:43:17.635ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
98842025-04-30T06:43:17.637ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2190f3b8-ff05-4196-b217-b39b4f6dbb18", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98852025-04-30T06:43:17.637ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
98862025-04-30T06:43:17.640ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c12b36d8-5edd-483a-a905-123b8e919435", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98872025-04-30T06:43:17.640ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
98882025-04-30T06:43:17.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00747258-587b-4c28-b636-f528e64c245b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98892025-04-30T06:43:17.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
98902025-04-30T06:43:17.646ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "667e4c53-bac9-4c28-98b7-018896b6f07c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98912025-04-30T06:43:17.646ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
98922025-04-30T06:43:17.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2810dd94-2276-4e76-b91a-d6b09073e7b9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98932025-04-30T06:43:17.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
98942025-04-30T06:43:17.659ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "520ec5c8-4579-4576-82ad-91f290ff09fd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98952025-04-30T06:43:17.659ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
98962025-04-30T06:43:17.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f47dceac-1cfa-4b90-a1f2-4ea5dba295f0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98972025-04-30T06:43:17.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
98982025-04-30T06:43:17.664ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a558d50f-23c4-49cc-b347-f091e01dadda", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
98992025-04-30T06:43:17.664ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
99002025-04-30T06:43:17.667ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc3a052b-8ec3-4aaf-bbcd-a0cf83ef146c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99012025-04-30T06:43:17.667ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
99022025-04-30T06:43:17.668ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96fe333f-e834-440b-9c67-15132e1818e1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99032025-04-30T06:43:17.669ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
99042025-04-30T06:43:17.670ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cae6870-7564-495c-a249-a6288db961e7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99052025-04-30T06:43:17.671ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
99062025-04-30T06:43:17.674ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e74c8369-4fea-48e1-857e-98f0f59ce4d8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99072025-04-30T06:43:17.674ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
99082025-04-30T06:43:17.677ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63204224-c4b0-4a50-8f37-143223fff207", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99092025-04-30T06:43:17.677ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
99102025-04-30T06:43:17.680ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b7c5c10-fc2a-45a2-985d-255fcb658e11", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99112025-04-30T06:43:17.680ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
99122025-04-30T06:43:17.681ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63d06e18-cb63-4609-ae9a-d9b5101248f5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99132025-04-30T06:43:17.682ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
99142025-04-30T06:43:17.682ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99152025-04-30T06:43:17.683ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99162025-04-30T06:43:17.683ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
99172025-04-30T06:43:17.685ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
99182025-04-30T06:43:17.686ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99192025-04-30T06:43:17.686ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99202025-04-30T06:43:17.686ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99212025-04-30T06:43:17.686ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
99222025-04-30T06:43:17.687ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99232025-04-30T06:43:17.688ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d904b68-59dd-4d87-82dc-f04204711a5c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99242025-04-30T06:43:17.688ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
99252025-04-30T06:43:17.690ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
99262025-04-30T06:43:17.690ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
99272025-04-30T06:43:17.690ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99282025-04-30T06:43:17.699ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99292025-04-30T06:43:17.700ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99302025-04-30T06:43:17.700ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
99312025-04-30T06:43:17.701ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99322025-04-30T06:43:17.703ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98c20db4-d968-422c-95b3-e493446c0fdc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99332025-04-30T06:43:17.703ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
99342025-04-30T06:43:17.705ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99352025-04-30T06:43:17.705ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
99362025-04-30T06:43:17.706ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99372025-04-30T06:43:17.707ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
99382025-04-30T06:43:17.707ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99392025-04-30T06:43:17.708ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b58b46ad-f9a4-4501-8ad5-2844319d6cf2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99402025-04-30T06:43:17.708ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
99412025-04-30T06:43:17.710ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b0046c5-c5e4-4104-a3e1-c3e6a6f34b6a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99422025-04-30T06:43:17.710ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
99432025-04-30T06:43:17.713ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ef73f9d-5a47-4e10-b963-2bf7c393b462", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99442025-04-30T06:43:17.713ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
99452025-04-30T06:43:17.715ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aa77977-a89b-4ae0-86ff-7c6fec01978f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99462025-04-30T06:43:17.715ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
99472025-04-30T06:43:17.717ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e52f07c9-9eb9-4ddf-8ff1-dc339855f641", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99482025-04-30T06:43:17.717ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
99492025-04-30T06:43:17.720ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87052bd5-e00d-4d3d-853b-fcd8903b9cfd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99502025-04-30T06:43:17.727ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
99512025-04-30T06:43:17.728ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99522025-04-30T06:43:17.728ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99532025-04-30T06:43:17.730ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99542025-04-30T06:43:17.730ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99552025-04-30T06:43:17.731ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99562025-04-30T06:43:17.731ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99572025-04-30T06:43:17.732ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "125db079-4e79-45ba-a30d-53101c9387ff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99582025-04-30T06:43:17.732ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
99592025-04-30T06:43:17.736ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e1cc585-3962-41f2-b52b-613f95441a38", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99602025-04-30T06:43:17.736ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
99612025-04-30T06:43:17.738ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6c4cfff-c66e-4230-898a-e3ab28799c35", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99622025-04-30T06:43:17.738ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
99632025-04-30T06:43:17.740ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "186506b1-4a07-48b8-b8da-65cb210fd05a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99642025-04-30T06:43:17.740ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
99652025-04-30T06:43:17.745ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82780239-1446-4e86-9669-ee177ede28b5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99662025-04-30T06:43:17.745ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
99672025-04-30T06:43:17.745ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
99682025-04-30T06:43:17.748ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "427c2373-69a3-4579-9afe-25b2c06927a5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99692025-04-30T06:43:17.748ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
99702025-04-30T06:43:17.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c88f977-f0be-44e5-8e07-4351adad6a19", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99712025-04-30T06:43:17.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
99722025-04-30T06:43:17.753ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e3aae42-8b29-4871-a03a-a6f0f62cb042", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99732025-04-30T06:43:17.754ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
99742025-04-30T06:43:17.756ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "634c452a-313a-4b91-a0f3-094d448789c9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99752025-04-30T06:43:17.756ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
99762025-04-30T06:43:17.759ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3a60367-ffdf-476d-9c5f-0ba6ad81290d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99772025-04-30T06:43:17.759ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
99782025-04-30T06:43:17.762ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c1485f1-f1e4-4924-b191-419fac40a823", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99792025-04-30T06:43:17.763ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
99802025-04-30T06:43:17.765ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9fe842f-c728-4696-aa99-03848455299a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99812025-04-30T06:43:17.765ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
99822025-04-30T06:43:17.769ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "175b2400-f83c-4b17-a955-cc74847d331c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99832025-04-30T06:43:17.769ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
99842025-04-30T06:43:17.770ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4edb4afc-aaee-4f29-9ce6-2be134145bfa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99852025-04-30T06:43:17.771ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
99862025-04-30T06:43:17.773ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d7f9729-9007-4691-aecd-c9e89a1914a3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99872025-04-30T06:43:17.773ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
99882025-04-30T06:43:17.775ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec6e0d2e-c0b6-4627-a712-726e6d8cdaf5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99892025-04-30T06:43:17.775ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
99902025-04-30T06:43:17.778ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d8ebef1-f11f-4716-872d-74037443a7d6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
99912025-04-30T06:43:17.778ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
99922025-04-30T06:43:17.778ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99932025-04-30T06:43:17.778ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99942025-04-30T06:43:17.778ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99952025-04-30T06:43:17.779ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99962025-04-30T06:43:17.781ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99972025-04-30T06:43:17.787ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99982025-04-30T06:43:17.787ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
99992025-04-30T06:43:17.787ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100002025-04-30T06:43:17.788ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100012025-04-30T06:43:17.788ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100022025-04-30T06:43:17.789ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100032025-04-30T06:43:17.791ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
100042025-04-30T06:43:17.792ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100052025-04-30T06:43:17.797ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
100062025-04-30T06:43:17.798ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100072025-04-30T06:43:17.798ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100082025-04-30T06:43:17.798ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100092025-04-30T06:43:17.805ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100102025-04-30T06:43:17.806ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100112025-04-30T06:43:17.806ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100122025-04-30T06:43:17.806ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100132025-04-30T06:43:17.806ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100142025-04-30T06:43:17.809ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100152025-04-30T06:43:17.816ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100162025-04-30T06:43:17.816ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100172025-04-30T06:43:17.816ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100182025-04-30T06:43:17.817ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100192025-04-30T06:43:17.871ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
100202025-04-30T06:43:17.871ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
100212025-04-30T06:43:17.872ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:58177 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
100222025-04-30T06:43:17.874ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100232025-04-30T06:43:17.874ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100242025-04-30T06:43:17.875ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100252025-04-30T06:43:17.875ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:58177 req_id = 30b1d085-b6a1-473a-9a00-4abe06f1e726 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
100262025-04-30T06:43:17.875ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 592 local_addr = [::1]:51119 method = GET remote_addr = [::1]:58177 req_id = 30b1d085-b6a1-473a-9a00-4abe06f1e726 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
100272025-04-30T06:43:17.876ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "30b1d085-b6a1-473a-9a00-4abe06f1e726", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
100282025-04-30T06:43:17.880ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
100292025-04-30T06:43:17.880ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
100302025-04-30T06:43:17.880ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
100312025-04-30T06:43:17.882ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
100322025-04-30T06:43:17.882ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100332025-04-30T06:43:17.882ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100342025-04-30T06:43:17.882ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100352025-04-30T06:43:17.883ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100362025-04-30T06:43:17.889ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100372025-04-30T06:43:17.896ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
100382025-04-30T06:43:17.896ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100392025-04-30T06:43:17.896ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100402025-04-30T06:43:17.897ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100412025-04-30T06:43:17.933ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
100422025-04-30T06:43:17.936ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
100432025-04-30T06:43:17.937ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
100442025-04-30T06:43:17.937ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 450.194303ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
100452025-04-30T06:43:17.937ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
100462025-04-30T06:43:17.937ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
100472025-04-30T06:43:17.937ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100482025-04-30T06:43:17.937ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100492025-04-30T06:43:17.937ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100502025-04-30T06:43:17.938ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100512025-04-30T06:43:17.940ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100522025-04-30T06:43:17.946ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
100532025-04-30T06:43:17.946ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100542025-04-30T06:43:17.946ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100552025-04-30T06:43:17.947ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100562025-04-30T06:43:17.950ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
100572025-04-30T06:43:17.950ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
100582025-04-30T06:43:17.950ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9b 55 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .U..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50755 req_id = 58412085-c89d-4a1c-9dc7-d723af604e06
100592025-04-30T06:43:17.950ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39765,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50755 req_id = 58412085-c89d-4a1c-9dc7-d723af604e06
100602025-04-30T06:43:17.951ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
100612025-04-30T06:43:17.951ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
100622025-04-30T06:43:17.951ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50755 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 58412085-c89d-4a1c-9dc7-d723af604e06
100632025-04-30T06:43:17.952ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4262, tv_nsec: 751743556 } })
100642025-04-30T06:43:17.952ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
100652025-04-30T06:43:17.953ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 32 1f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 2...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54586 req_id = 81052271-037b-46e0-9c6a-d9b940ced606
100662025-04-30T06:43:17.953ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12831,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54586 req_id = 81052271-037b-46e0-9c6a-d9b940ced606
100672025-04-30T06:43:17.953ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
100682025-04-30T06:43:17.954ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54586 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 81052271-037b-46e0-9c6a-d9b940ced606
100692025-04-30T06:43:18.016ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
100702025-04-30T06:43:18.017ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100712025-04-30T06:43:18.017ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100722025-04-30T06:43:18.017ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100732025-04-30T06:43:18.017ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100742025-04-30T06:43:18.018ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100752025-04-30T06:43:18.019ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5e c7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ^............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59434 req_id = 5fafe599-66c0-4f11-ad97-e20d46436d1b
100762025-04-30T06:43:18.020ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24263,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59434 req_id = 5fafe599-66c0-4f11-ad97-e20d46436d1b
100772025-04-30T06:43:18.020ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
100782025-04-30T06:43:18.020ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24263, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59434 req_id = 5fafe599-66c0-4f11-ad97-e20d46436d1b
100792025-04-30T06:43:18.022ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
100802025-04-30T06:43:18.022ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
100812025-04-30T06:43:18.022ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
100822025-04-30T06:43:18.024ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100832025-04-30T06:43:18.035ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100842025-04-30T06:43:18.035ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100852025-04-30T06:43:18.036ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100862025-04-30T06:43:18.036ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100872025-04-30T06:43:18.047ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100882025-04-30T06:43:18.047ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100892025-04-30T06:43:18.049ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100902025-04-30T06:43:18.051ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1f01683-edae-4060-bf9f-40065351d127", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
100912025-04-30T06:43:18.052ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
100922025-04-30T06:43:18.069ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df7ef71d-77a5-49cf-b949-58169abf7c13", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
100932025-04-30T06:43:18.070ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
100942025-04-30T06:43:18.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62af2aeb-456a-4790-be41-5c3178f8f8c6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
100952025-04-30T06:43:18.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
100962025-04-30T06:43:18.085ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96af451b-0dc1-446e-8187-bef0609e5e17", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
100972025-04-30T06:43:18.085ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
100982025-04-30T06:43:18.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac4841d-fa16-45b7-9d97-e9b2a4fd6a07", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
100992025-04-30T06:43:18.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
101002025-04-30T06:43:18.098ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e08c36-a5ae-4489-b02b-8390f778366d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101012025-04-30T06:43:18.098ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
101022025-04-30T06:43:18.105ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
101032025-04-30T06:43:18.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afb44bb1-4d21-4a46-96ae-5e9d6fded51f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101042025-04-30T06:43:18.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
101052025-04-30T06:43:18.111ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33f55a64-8568-4bd3-b48b-9cf47f0e63f2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101062025-04-30T06:43:18.111ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
101072025-04-30T06:43:18.113ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "484effb7-dec0-49c2-96ed-6a8df999f092", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101082025-04-30T06:43:18.113ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
101092025-04-30T06:43:18.114ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83e5e516-c067-450a-96f8-6f0a4cf87990", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101102025-04-30T06:43:18.114ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
101112025-04-30T06:43:18.115ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dfd9fbd-bf11-4283-91e9-984fd0284d0f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101122025-04-30T06:43:18.115ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
101132025-04-30T06:43:18.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bd747d5-9c93-431c-9b34-d5f773f63b7a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101142025-04-30T06:43:18.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
101152025-04-30T06:43:18.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7427ff1-af30-4cc4-87cb-0b3246f43dbd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101162025-04-30T06:43:18.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
101172025-04-30T06:43:18.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d61fec1-aa11-4f77-a099-ff8dbccd1dbd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101182025-04-30T06:43:18.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
101192025-04-30T06:43:18.125ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6019ea8-165f-48d7-9568-dd53acd7663b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101202025-04-30T06:43:18.125ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
101212025-04-30T06:43:18.127ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05b4366b-c351-4d26-b6cd-02dfdd7774cc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101222025-04-30T06:43:18.127ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
101232025-04-30T06:43:18.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf5971be-6425-43af-b811-16a5714437a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101242025-04-30T06:43:18.129ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
101252025-04-30T06:43:18.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69126315-2ad9-4633-8bd2-a886fe6b0963", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101262025-04-30T06:43:18.132ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
101272025-04-30T06:43:18.134ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0693f160-3dc0-4fb0-8797-ef6a540e6a70", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101282025-04-30T06:43:18.135ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
101292025-04-30T06:43:18.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8f2e9f2-025a-4e6a-b99c-64b52b420394", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101302025-04-30T06:43:18.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
101312025-04-30T06:43:18.138ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1bbebf7-ce35-403f-a384-820e803d7107", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101322025-04-30T06:43:18.139ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
101332025-04-30T06:43:18.141ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5485ed3-8f27-4fc1-8e12-723744ff98ef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101342025-04-30T06:43:18.142ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
101352025-04-30T06:43:18.142ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5d63145-9e9b-480e-a4ba-97fa4c362742", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101362025-04-30T06:43:18.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
101372025-04-30T06:43:18.144ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d91c94e-67b1-4b65-9171-9928e749c1a2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101382025-04-30T06:43:18.144ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
101392025-04-30T06:43:18.147ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "815d0cac-55a9-482f-bfed-3feb13588fe6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101402025-04-30T06:43:18.147ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
101412025-04-30T06:43:18.158ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101422025-04-30T06:43:18.158ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101432025-04-30T06:43:18.158ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101442025-04-30T06:43:18.158ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
101452025-04-30T06:43:18.159ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101462025-04-30T06:43:18.160ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b601205c-8bd4-41b3-b939-30a4371b84cd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101472025-04-30T06:43:18.160ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
101482025-04-30T06:43:18.163ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101492025-04-30T06:43:18.169ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101502025-04-30T06:43:18.169ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101512025-04-30T06:43:18.169ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
101522025-04-30T06:43:18.170ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101532025-04-30T06:43:18.172ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26277874-d510-4eb9-962a-adf89ee243e4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101542025-04-30T06:43:18.172ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
101552025-04-30T06:43:18.172ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101562025-04-30T06:43:18.173ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
101572025-04-30T06:43:18.173ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101582025-04-30T06:43:18.175ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00696291-6155-4d5e-bffe-914f73a74adc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101592025-04-30T06:43:18.175ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
101602025-04-30T06:43:18.180ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8edf7ca5-e616-4b64-a9f8-0f89e534c83a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101612025-04-30T06:43:18.180ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
101622025-04-30T06:43:18.184ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41a319ce-e9a0-4a87-9acc-db2291e3643b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101632025-04-30T06:43:18.184ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
101642025-04-30T06:43:18.184ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36259206-5818-4410-8f64-e6813d64629d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101652025-04-30T06:43:18.184ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
101662025-04-30T06:43:18.190ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecde847e-6f2d-42d1-b573-6750f16c8544", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101672025-04-30T06:43:18.193ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
101682025-04-30T06:43:18.195ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "060cb3ee-ee82-4105-8800-89d972eef488", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101692025-04-30T06:43:18.195ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
101702025-04-30T06:43:18.197ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d40512b8-3203-405e-9103-fe005a02cba9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101712025-04-30T06:43:18.197ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
101722025-04-30T06:43:18.200ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07a43402-6e47-4315-a9e5-a1f7c694731e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101732025-04-30T06:43:18.201ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
101742025-04-30T06:43:18.206ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62c64e35-1cc1-409a-a87e-ff815175fb1b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101752025-04-30T06:43:18.206ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
101762025-04-30T06:43:18.211ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e29eee7-ddbb-45a0-b716-9da77acf3e9c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101772025-04-30T06:43:18.211ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
101782025-04-30T06:43:18.220ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5632543f-8f3a-4d9e-9dfd-f824cb4f8ff8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101792025-04-30T06:43:18.220ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
101802025-04-30T06:43:18.222ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f6914fe-3841-4968-ace3-61bf0d404d4d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101812025-04-30T06:43:18.222ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
101822025-04-30T06:43:18.225ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "596fdc61-b53a-47b8-9bb1-4bf3aa0a43fe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101832025-04-30T06:43:18.225ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
101842025-04-30T06:43:18.227ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f4d8c9c-c928-48a5-9a66-1f3859a23a09", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101852025-04-30T06:43:18.227ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
101862025-04-30T06:43:18.229ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb582ddf-179b-4eaf-bc55-a03ed194c636", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101872025-04-30T06:43:18.230ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
101882025-04-30T06:43:18.231ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac6226e3-7855-4c82-87a3-5970322319a9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101892025-04-30T06:43:18.231ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
101902025-04-30T06:43:18.232ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91994867-79fe-4796-a17c-453b0c818a7b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101912025-04-30T06:43:18.233ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
101922025-04-30T06:43:18.233ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bbf9d22-9706-4945-9b87-73e52d54736e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
101932025-04-30T06:43:18.234ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101942025-04-30T06:43:18.234ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101952025-04-30T06:43:18.234ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101962025-04-30T06:43:18.234ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101972025-04-30T06:43:18.234ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101982025-04-30T06:43:18.241ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
101992025-04-30T06:43:18.247ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102002025-04-30T06:43:18.252ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
102012025-04-30T06:43:18.252ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
102022025-04-30T06:43:18.252ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102032025-04-30T06:43:18.252ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102042025-04-30T06:43:18.253ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102052025-04-30T06:43:18.254ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102062025-04-30T06:43:18.271ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102072025-04-30T06:43:18.278ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
102082025-04-30T06:43:18.278ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102092025-04-30T06:43:18.278ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102102025-04-30T06:43:18.278ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102112025-04-30T06:43:18.368ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
102122025-04-30T06:43:18.368ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
102132025-04-30T06:43:18.368ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102142025-04-30T06:43:18.368ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102152025-04-30T06:43:18.368ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102162025-04-30T06:43:18.368ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102172025-04-30T06:43:18.369ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102182025-04-30T06:43:18.369ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:54160 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
102192025-04-30T06:43:18.370ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:54160 req_id = 19196377-e4e7-4365-bf92-88ffab9f2b57 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
102202025-04-30T06:43:18.370ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 500 local_addr = [::1]:51119 method = GET remote_addr = [::1]:54160 req_id = 19196377-e4e7-4365-bf92-88ffab9f2b57 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
102212025-04-30T06:43:18.370ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "19196377-e4e7-4365-bf92-88ffab9f2b57", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
102222025-04-30T06:43:18.373ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
102232025-04-30T06:43:18.373ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
102242025-04-30T06:43:18.373ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
102252025-04-30T06:43:18.375ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
102262025-04-30T06:43:18.375ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102272025-04-30T06:43:18.375ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102282025-04-30T06:43:18.375ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102292025-04-30T06:43:18.375ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102302025-04-30T06:43:18.376ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102312025-04-30T06:43:18.383ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102322025-04-30T06:43:18.383ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102332025-04-30T06:43:18.383ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102342025-04-30T06:43:18.384ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102352025-04-30T06:43:18.386ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102362025-04-30T06:43:18.393ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
102372025-04-30T06:43:18.396ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102382025-04-30T06:43:18.396ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102392025-04-30T06:43:18.397ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102402025-04-30T06:43:18.399ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102412025-04-30T06:43:18.400ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102422025-04-30T06:43:18.400ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102432025-04-30T06:43:18.433ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
102442025-04-30T06:43:18.434ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
102452025-04-30T06:43:18.434ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 497.196282ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
102462025-04-30T06:43:18.434ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
102472025-04-30T06:43:18.434ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
102482025-04-30T06:43:18.434ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102492025-04-30T06:43:18.434ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102502025-04-30T06:43:18.434ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102512025-04-30T06:43:18.437ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102522025-04-30T06:43:18.439ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102532025-04-30T06:43:18.442ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
102542025-04-30T06:43:18.442ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102552025-04-30T06:43:18.443ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102562025-04-30T06:43:18.443ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102572025-04-30T06:43:18.448ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
102582025-04-30T06:43:18.449ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
102592025-04-30T06:43:18.449ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
102602025-04-30T06:43:18.450ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 05 92 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61119 req_id = 0feef7c6-897c-4e24-b823-42c3f03d8dc5
102612025-04-30T06:43:18.450ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1426,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61119 req_id = 0feef7c6-897c-4e24-b823-42c3f03d8dc5
102622025-04-30T06:43:18.450ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
102632025-04-30T06:43:18.450ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
102642025-04-30T06:43:18.450ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61119 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0feef7c6-897c-4e24-b823-42c3f03d8dc5
102652025-04-30T06:43:18.450ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4263, tv_nsec: 249736454 } })
102662025-04-30T06:43:18.450ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
102672025-04-30T06:43:18.451ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 36 3d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 6=..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51441 req_id = 1a80f897-fcfc-4c58-b41c-cb220ba7dc53
102682025-04-30T06:43:18.451ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13885,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51441 req_id = 1a80f897-fcfc-4c58-b41c-cb220ba7dc53
102692025-04-30T06:43:18.451ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
102702025-04-30T06:43:18.451ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51441 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1a80f897-fcfc-4c58-b41c-cb220ba7dc53
102712025-04-30T06:43:18.519ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
102722025-04-30T06:43:18.520ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102732025-04-30T06:43:18.520ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102742025-04-30T06:43:18.520ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102752025-04-30T06:43:18.520ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102762025-04-30T06:43:18.521ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102772025-04-30T06:43:18.521ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 03 14 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62853 req_id = d505d6e3-8b3b-4d2a-bb4f-e9a066571ab1
102782025-04-30T06:43:18.521ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 788,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62853 req_id = d505d6e3-8b3b-4d2a-bb4f-e9a066571ab1
102792025-04-30T06:43:18.521ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
102802025-04-30T06:43:18.521ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 788, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62853 req_id = d505d6e3-8b3b-4d2a-bb4f-e9a066571ab1
102812025-04-30T06:43:18.522ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
102822025-04-30T06:43:18.522ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
102832025-04-30T06:43:18.522ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
102842025-04-30T06:43:18.525ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102852025-04-30T06:43:18.532ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102862025-04-30T06:43:18.532ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102872025-04-30T06:43:18.532ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102882025-04-30T06:43:18.532ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102892025-04-30T06:43:18.535ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e39370f5-8e9f-4d29-a5e0-85022ad7f6c7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
102902025-04-30T06:43:18.535ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
102912025-04-30T06:43:18.537ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102922025-04-30T06:43:18.537ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102932025-04-30T06:43:18.538ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102942025-04-30T06:43:18.539ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43ae844a-f68a-44df-bec3-5e7e5caf00fb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
102952025-04-30T06:43:18.540ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
102962025-04-30T06:43:18.542ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3237a563-805e-47e8-ae4f-9c56c07af393", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
102972025-04-30T06:43:18.542ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
102982025-04-30T06:43:18.544ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "091850df-4184-42c4-88a5-e60862b88796", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
102992025-04-30T06:43:18.545ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
103002025-04-30T06:43:18.549ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9da444bc-5dc3-4377-a71e-787476e36421", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103012025-04-30T06:43:18.549ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
103022025-04-30T06:43:18.553ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfaa70c0-38db-45f5-82e6-a1d6cc11698b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103032025-04-30T06:43:18.553ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
103042025-04-30T06:43:18.556ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca6c407-0626-4a81-92d0-85031f94e114", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103052025-04-30T06:43:18.556ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
103062025-04-30T06:43:18.559ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1695d5fa-5494-4344-a738-507c1cfcd5d7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103072025-04-30T06:43:18.559ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
103082025-04-30T06:43:18.561ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d07587e9-41c6-4904-aeb0-1055f38197e8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103092025-04-30T06:43:18.561ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
103102025-04-30T06:43:18.562ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fec4bc5d-829d-477b-927b-55300d1e600b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103112025-04-30T06:43:18.562ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
103122025-04-30T06:43:18.563ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43dcee14-5e52-4408-a76d-389d2e46486c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103132025-04-30T06:43:18.563ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
103142025-04-30T06:43:18.564ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b2fcc67-8eba-49ce-9266-17bbc8491a8f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103152025-04-30T06:43:18.565ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
103162025-04-30T06:43:18.569ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
103172025-04-30T06:43:18.569ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d14d442-88b6-4857-a098-6e071918cbc7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103182025-04-30T06:43:18.569ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
103192025-04-30T06:43:18.571ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "072a5d56-9990-4a16-aaeb-fb685fc94431", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
103202025-04-30T06:43:18.571ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
103212025-04-30T06:43:18.581ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06040835-82c7-4691-b2ab-d0fc9228d40c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103222025-04-30T06:43:18.581ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
103232025-04-30T06:43:18.585ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db9a7f21-1502-4bf2-a030-e2f62472b273", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103242025-04-30T06:43:18.585ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
103252025-04-30T06:43:18.587ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b1ffc08-47ea-4da8-8554-31882ae56b94", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103262025-04-30T06:43:18.587ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
103272025-04-30T06:43:18.589ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e624f16a-6537-4383-b3e8-0336b694213a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103282025-04-30T06:43:18.589ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
103292025-04-30T06:43:18.591ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbefd5fc-64ea-41c4-a98e-608a390e7f02", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103302025-04-30T06:43:18.591ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
103312025-04-30T06:43:18.592ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3007a299-bca4-4332-b534-dbe897776d71", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103322025-04-30T06:43:18.592ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
103332025-04-30T06:43:18.594ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e68028f8-c008-4407-ac81-c46075df6be7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103342025-04-30T06:43:18.594ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
103352025-04-30T06:43:18.595ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73cd98f1-bea9-4f09-a026-bb5451b2c718", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103362025-04-30T06:43:18.595ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
103372025-04-30T06:43:18.596ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "652f2804-0e60-4944-952d-d2c5690952c1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103382025-04-30T06:43:18.596ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
103392025-04-30T06:43:18.597ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67aba1a9-e825-49ec-8338-8fc438143b52", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103402025-04-30T06:43:18.598ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
103412025-04-30T06:43:18.600ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d24f3100-91ab-4fea-a934-a39e61e18510", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103422025-04-30T06:43:18.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
103432025-04-30T06:43:18.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "765ba405-ef93-46d3-9d32-709976ec1608", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103442025-04-30T06:43:18.603ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
103452025-04-30T06:43:18.604ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "965f9968-8da4-47d7-90cb-00104fd51648", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103462025-04-30T06:43:18.604ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
103472025-04-30T06:43:18.607ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f34d1807-d290-4279-b2da-3ddfa997ad23", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103482025-04-30T06:43:18.608ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
103492025-04-30T06:43:18.615ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9e7c492-7fbd-4bfc-b4ad-e19f08a2f14d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103502025-04-30T06:43:18.615ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
103512025-04-30T06:43:18.618ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a02abd91-7284-4a80-9e21-e703355507fd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103522025-04-30T06:43:18.618ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
103532025-04-30T06:43:18.619ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9d3c975-a741-4bcd-b520-0458a502d047", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103542025-04-30T06:43:18.619ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
103552025-04-30T06:43:18.621ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103562025-04-30T06:43:18.621ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103572025-04-30T06:43:18.621ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103582025-04-30T06:43:18.621ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
103592025-04-30T06:43:18.621ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103602025-04-30T06:43:18.622ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "263c2a67-379e-45dd-bba9-a19999382250", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103612025-04-30T06:43:18.622ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
103622025-04-30T06:43:18.625ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103632025-04-30T06:43:18.631ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103642025-04-30T06:43:18.631ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103652025-04-30T06:43:18.631ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
103662025-04-30T06:43:18.631ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103672025-04-30T06:43:18.632ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "acb1fd0f-6eb7-4d88-8484-ea2b24d8d51d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103682025-04-30T06:43:18.632ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
103692025-04-30T06:43:18.634ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "effd161b-e85d-4531-85db-ccc3a3c180b1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103702025-04-30T06:43:18.634ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
103712025-04-30T06:43:18.634ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103722025-04-30T06:43:18.634ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
103732025-04-30T06:43:18.635ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103742025-04-30T06:43:18.636ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2399dd48-c31c-4f3a-a071-f21d009969a9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103752025-04-30T06:43:18.637ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
103762025-04-30T06:43:18.639ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b81f149d-c16c-42e6-90b0-4cead99f8668", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103772025-04-30T06:43:18.639ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
103782025-04-30T06:43:18.642ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24acc26e-b7b1-4a14-87cd-0647b93c0ff2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103792025-04-30T06:43:18.642ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
103802025-04-30T06:43:18.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2f99fbe-6362-4acc-b234-63869b8f185d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103812025-04-30T06:43:18.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
103822025-04-30T06:43:18.647ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "785cb49b-3932-4ee3-b757-42288c77da05", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103832025-04-30T06:43:18.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
103842025-04-30T06:43:18.654ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4dad0cc-1b32-4807-bcda-a15f4a352f49", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103852025-04-30T06:43:18.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
103862025-04-30T06:43:18.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba893593-3e79-4ac9-ac8a-bcc7b9cba26a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103872025-04-30T06:43:18.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
103882025-04-30T06:43:18.660ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4c00d38-8304-4ed6-9b78-176966810f01", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103892025-04-30T06:43:18.660ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
103902025-04-30T06:43:18.663ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b658c87e-36e7-4fae-a45f-a794c077279d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103912025-04-30T06:43:18.664ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
103922025-04-30T06:43:18.669ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "847dbcfa-010d-460c-a3cd-e2dd11f9e17b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103932025-04-30T06:43:18.670ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
103942025-04-30T06:43:18.673ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce627c94-8515-4fcf-a624-6d39664a415e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
103952025-04-30T06:43:18.673ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103962025-04-30T06:43:18.673ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103972025-04-30T06:43:18.673ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103982025-04-30T06:43:18.673ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103992025-04-30T06:43:18.674ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104002025-04-30T06:43:18.677ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104012025-04-30T06:43:18.682ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
104022025-04-30T06:43:18.682ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
104032025-04-30T06:43:18.682ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104042025-04-30T06:43:18.682ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104052025-04-30T06:43:18.682ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104062025-04-30T06:43:18.683ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104072025-04-30T06:43:18.683ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
104082025-04-30T06:43:18.684ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
104092025-04-30T06:43:18.684ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
104102025-04-30T06:43:18.684ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
104112025-04-30T06:43:18.685ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
104122025-04-30T06:43:18.687ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104132025-04-30T06:43:18.692ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
104142025-04-30T06:43:18.692ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104152025-04-30T06:43:18.692ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104162025-04-30T06:43:18.692ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104172025-04-30T06:43:18.695ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
104182025-04-30T06:43:18.695ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
104192025-04-30T06:43:18.741ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
104202025-04-30T06:43:18.741ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
104212025-04-30T06:43:18.741ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104222025-04-30T06:43:18.741ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104232025-04-30T06:43:18.741ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104242025-04-30T06:43:18.741ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
104252025-04-30T06:43:18.742ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104262025-04-30T06:43:18.743ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:47082 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
104272025-04-30T06:43:18.743ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104282025-04-30T06:43:18.743ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104292025-04-30T06:43:18.744ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:47082 req_id = 8b53e84c-6ea3-4e77-b6bc-5662e9b775b7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
104302025-04-30T06:43:18.753ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 2788 local_addr = [::1]:51119 method = GET remote_addr = [::1]:47082 req_id = 8b53e84c-6ea3-4e77-b6bc-5662e9b775b7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
104312025-04-30T06:43:18.754ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
104322025-04-30T06:43:18.754ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
104332025-04-30T06:43:18.755ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104342025-04-30T06:43:18.756ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104352025-04-30T06:43:18.756ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104362025-04-30T06:43:18.756ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104372025-04-30T06:43:18.756ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b53e84c-6ea3-4e77-b6bc-5662e9b775b7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
104382025-04-30T06:43:18.759ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
104392025-04-30T06:43:18.759ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
104402025-04-30T06:43:18.759ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
104412025-04-30T06:43:18.762ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
104422025-04-30T06:43:18.762ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104432025-04-30T06:43:18.762ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104442025-04-30T06:43:18.762ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104452025-04-30T06:43:18.763ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104462025-04-30T06:43:18.765ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104472025-04-30T06:43:18.772ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104482025-04-30T06:43:18.772ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104492025-04-30T06:43:18.772ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
104502025-04-30T06:43:18.773ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104512025-04-30T06:43:18.775ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104522025-04-30T06:43:18.780ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104532025-04-30T06:43:18.780ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104542025-04-30T06:43:18.780ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104552025-04-30T06:43:18.781ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104562025-04-30T06:43:18.782ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104572025-04-30T06:43:18.782ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
104582025-04-30T06:43:18.783ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104592025-04-30T06:43:18.785ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
104602025-04-30T06:43:18.810ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
104612025-04-30T06:43:18.811ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
104622025-04-30T06:43:18.811ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 377.19223ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
104632025-04-30T06:43:18.811ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
104642025-04-30T06:43:18.811ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
104652025-04-30T06:43:18.811ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104662025-04-30T06:43:18.811ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104672025-04-30T06:43:18.811ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104682025-04-30T06:43:18.812ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104692025-04-30T06:43:18.816ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
104702025-04-30T06:43:18.817ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104712025-04-30T06:43:18.822ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104722025-04-30T06:43:18.822ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104732025-04-30T06:43:18.822ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104742025-04-30T06:43:18.823ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104752025-04-30T06:43:18.828ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
104762025-04-30T06:43:18.828ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
104772025-04-30T06:43:18.830ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 82 9a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55014 req_id = 6e486f1f-dad1-4301-8db4-3aceb250ebc2
104782025-04-30T06:43:18.830ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33434,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55014 req_id = 6e486f1f-dad1-4301-8db4-3aceb250ebc2
104792025-04-30T06:43:18.830ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
104802025-04-30T06:43:18.831ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104812025-04-30T06:43:18.831ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55014 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6e486f1f-dad1-4301-8db4-3aceb250ebc2
104822025-04-30T06:43:18.832ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4263, tv_nsec: 631205146 } })
104832025-04-30T06:43:18.832ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
104842025-04-30T06:43:18.833ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 39 b2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 9...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49207 req_id = a3c930fe-4651-48af-af40-fee733418ab9
104852025-04-30T06:43:18.833ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14770,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49207 req_id = a3c930fe-4651-48af-af40-fee733418ab9
104862025-04-30T06:43:18.833ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104872025-04-30T06:43:18.833ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49207 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a3c930fe-4651-48af-af40-fee733418ab9
104882025-04-30T06:43:18.887ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
104892025-04-30T06:43:18.887ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104902025-04-30T06:43:18.887ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104912025-04-30T06:43:18.887ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104922025-04-30T06:43:18.888ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
104932025-04-30T06:43:18.889ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104942025-04-30T06:43:18.890ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 72 52 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c rR..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57785 req_id = 681ac64d-6b5c-4c97-b4bd-2efc428416ae
104952025-04-30T06:43:18.890ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29266,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57785 req_id = 681ac64d-6b5c-4c97-b4bd-2efc428416ae
104962025-04-30T06:43:18.890ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
104972025-04-30T06:43:18.890ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29266, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57785 req_id = 681ac64d-6b5c-4c97-b4bd-2efc428416ae
104982025-04-30T06:43:18.890ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
104992025-04-30T06:43:18.891ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
105002025-04-30T06:43:18.891ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
105012025-04-30T06:43:18.894ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105022025-04-30T06:43:18.901ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105032025-04-30T06:43:18.901ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105042025-04-30T06:43:18.901ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
105052025-04-30T06:43:18.902ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105062025-04-30T06:43:18.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee99e759-4b75-4635-87d6-706470d6d951", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105072025-04-30T06:43:18.909ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
105082025-04-30T06:43:18.910ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105092025-04-30T06:43:18.911ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
105102025-04-30T06:43:18.911ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105112025-04-30T06:43:18.915ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6d9dbfc-dc1d-442f-862e-15be5ee75b02", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105122025-04-30T06:43:18.915ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
105132025-04-30T06:43:18.921ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c03c9632-5c43-4c59-a87a-aed8febbe016", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105142025-04-30T06:43:18.921ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
105152025-04-30T06:43:18.925ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcad7f60-d5b4-448d-a11e-0f32aa30d36d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105162025-04-30T06:43:18.925ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
105172025-04-30T06:43:18.929ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d96c3c79-c151-433a-846a-5afbbc25b24c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105182025-04-30T06:43:18.929ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
105192025-04-30T06:43:18.934ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "280234f8-a697-4f88-8f39-35d601865cbe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105202025-04-30T06:43:18.934ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
105212025-04-30T06:43:18.943ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fcaa6d0-8417-48b8-b477-c7901b61e09a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105222025-04-30T06:43:18.943ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
105232025-04-30T06:43:18.944ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96971e05-00ea-4c7f-859c-2dcf79076fa1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105242025-04-30T06:43:18.944ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
105252025-04-30T06:43:18.944ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
105262025-04-30T06:43:18.944ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a1e89a5-3a75-4ebc-9125-14d61df55924", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105272025-04-30T06:43:18.944ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
105282025-04-30T06:43:18.944ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "545d317b-0356-4aaa-97d7-d00559853bdd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105292025-04-30T06:43:18.945ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
105302025-04-30T06:43:18.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ed187fb-387c-4d57-8741-fda3370fec1a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105312025-04-30T06:43:18.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
105322025-04-30T06:43:18.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be5f5baf-1d0c-4496-9ee7-3b42cf040128", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105332025-04-30T06:43:18.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
105342025-04-30T06:43:18.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ea8988e-0cd5-4f5b-b5e5-54b6c105883f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105352025-04-30T06:43:18.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
105362025-04-30T06:43:18.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8905d01b-09ec-4172-83fb-047ba7e5f703", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105372025-04-30T06:43:18.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
105382025-04-30T06:43:18.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40b1a523-2024-4d36-a45f-e3ff5814196a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105392025-04-30T06:43:18.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
105402025-04-30T06:43:18.957ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b485f178-8a15-4070-902e-73297fc9d556", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105412025-04-30T06:43:18.957ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
105422025-04-30T06:43:18.961ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "840e75b3-f6e0-46ad-a4b6-80cc6de76fb8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105432025-04-30T06:43:18.961ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
105442025-04-30T06:43:18.964ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c4be7ff-abcf-456b-840a-458278d0c244", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105452025-04-30T06:43:18.964ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
105462025-04-30T06:43:18.971ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02904bf3-3a58-4036-9166-d6290582b146", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105472025-04-30T06:43:18.972ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
105482025-04-30T06:43:18.976ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fe6af24-41be-4165-92ca-9228196e88b0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105492025-04-30T06:43:18.976ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
105502025-04-30T06:43:18.978ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fca20080-45ad-4216-8acc-165adf435e02", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105512025-04-30T06:43:18.978ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
105522025-04-30T06:43:18.981ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ede91d7-8e99-4ff0-9c67-f5912c89bd45", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105532025-04-30T06:43:18.982ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
105542025-04-30T06:43:18.987ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d394c46d-5004-421e-bd85-b57de888bdb6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105552025-04-30T06:43:18.988ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
105562025-04-30T06:43:18.995ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105572025-04-30T06:43:18.996ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105582025-04-30T06:43:18.996ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105592025-04-30T06:43:18.996ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
105602025-04-30T06:43:18.997ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105612025-04-30T06:43:19.017ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105622025-04-30T06:43:19.025ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105632025-04-30T06:43:19.025ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105642025-04-30T06:43:19.025ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
105652025-04-30T06:43:19.026ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105662025-04-30T06:43:19.031ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105672025-04-30T06:43:19.032ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
105682025-04-30T06:43:19.033ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105692025-04-30T06:43:19.039ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c518c1f2-32b4-4707-ab96-722f51bb32bf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105702025-04-30T06:43:19.039ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
105712025-04-30T06:43:19.057ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e2aa3b0-c071-4162-8aff-0d7741904fb3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105722025-04-30T06:43:19.057ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
105732025-04-30T06:43:19.060ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35d6aef5-31b1-49ad-9db5-a578ce0285e8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105742025-04-30T06:43:19.061ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
105752025-04-30T06:43:19.066ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "384d2eee-7228-4cc6-bc41-24f93c72f7e0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105762025-04-30T06:43:19.066ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
105772025-04-30T06:43:19.073ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c60ea6b7-6b08-4b0a-9297-d17308d0ebbf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105782025-04-30T06:43:19.073ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
105792025-04-30T06:43:19.074ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
105802025-04-30T06:43:19.079ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b09704da-fb00-476e-9bdb-5a81bc7cc634", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105812025-04-30T06:43:19.079ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
105822025-04-30T06:43:19.084ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2128ffaa-d623-4814-a621-1dd6e04bb820", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105832025-04-30T06:43:19.085ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
105842025-04-30T06:43:19.085ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4931d215-eb5d-4e44-9822-4273764ceaec", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105852025-04-30T06:43:19.086ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
105862025-04-30T06:43:19.087ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f55ed221-f650-46b0-becd-510616c5cd4d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105872025-04-30T06:43:19.087ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
105882025-04-30T06:43:19.088ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22d11b80-c71d-418c-aef3-17f9295946dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105892025-04-30T06:43:19.088ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
105902025-04-30T06:43:19.090ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "621d484a-eca8-463d-9e1c-6b7ba3fede44", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105912025-04-30T06:43:19.091ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
105922025-04-30T06:43:19.093ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b854daf-1eb1-426f-ad92-07aa95c43dee", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105932025-04-30T06:43:19.093ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
105942025-04-30T06:43:19.095ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ce428f5-9db5-48e3-8bc7-8ebaed6df004", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105952025-04-30T06:43:19.095ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
105962025-04-30T06:43:19.098ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2265b6c8-0651-4f6e-88f7-1d832045f94d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105972025-04-30T06:43:19.098ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
105982025-04-30T06:43:19.100ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1e6e24c-258a-4309-89d7-77e690853b05", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
105992025-04-30T06:43:19.100ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
106002025-04-30T06:43:19.104ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fae6a844-9e8c-4d2f-92ac-4b5ea89e3bfc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
106012025-04-30T06:43:19.104ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
106022025-04-30T06:43:19.107ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82b30791-2e32-43c9-9ea4-22cb5a36eef1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
106032025-04-30T06:43:19.107ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
106042025-04-30T06:43:19.109ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ecb8bc2-20fe-4d07-9a0f-48736baa3ae3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
106052025-04-30T06:43:19.109ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
106062025-04-30T06:43:19.111ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a2b8fa-8321-4d02-bc8b-d4c7bfed5683", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
106072025-04-30T06:43:19.111ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
106082025-04-30T06:43:19.114ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31da591e-83a6-4ad5-bb80-183544984a78", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
106092025-04-30T06:43:19.114ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
106102025-04-30T06:43:19.115ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "463b33f3-7bed-48fd-89a2-3416d15653db", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
106112025-04-30T06:43:19.115ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
106122025-04-30T06:43:19.116ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de1a5d6a-d611-4ba3-8d02-a145009083c4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
106132025-04-30T06:43:19.116ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
106142025-04-30T06:43:19.116ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106152025-04-30T06:43:19.116ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106162025-04-30T06:43:19.116ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106172025-04-30T06:43:19.117ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106182025-04-30T06:43:19.120ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106192025-04-30T06:43:19.128ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
106202025-04-30T06:43:19.128ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
106212025-04-30T06:43:19.128ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106222025-04-30T06:43:19.128ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106232025-04-30T06:43:19.128ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106242025-04-30T06:43:19.128ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106252025-04-30T06:43:19.129ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106262025-04-30T06:43:19.129ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106272025-04-30T06:43:19.129ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106282025-04-30T06:43:19.129ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
106292025-04-30T06:43:19.130ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106302025-04-30T06:43:19.131ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106312025-04-30T06:43:19.136ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
106322025-04-30T06:43:19.136ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106332025-04-30T06:43:19.136ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106342025-04-30T06:43:19.137ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106352025-04-30T06:43:19.138ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106362025-04-30T06:43:19.146ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106372025-04-30T06:43:19.146ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106382025-04-30T06:43:19.146ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
106392025-04-30T06:43:19.147ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106402025-04-30T06:43:19.158ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106412025-04-30T06:43:19.158ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
106422025-04-30T06:43:19.160ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106432025-04-30T06:43:19.232ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
106442025-04-30T06:43:19.232ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
106452025-04-30T06:43:19.234ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:44037 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
106462025-04-30T06:43:19.236ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:44037 req_id = acaa14c5-b460-4394-8000-6447adc54571 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106472025-04-30T06:43:19.237ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 1165 local_addr = [::1]:51119 method = GET remote_addr = [::1]:44037 req_id = acaa14c5-b460-4394-8000-6447adc54571 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106482025-04-30T06:43:19.238ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "acaa14c5-b460-4394-8000-6447adc54571", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
106492025-04-30T06:43:19.242ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106502025-04-30T06:43:19.242ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106512025-04-30T06:43:19.242ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
106522025-04-30T06:43:19.244ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106532025-04-30T06:43:19.244ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106542025-04-30T06:43:19.244ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106552025-04-30T06:43:19.244ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106562025-04-30T06:43:19.245ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106572025-04-30T06:43:19.267ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106582025-04-30T06:43:19.272ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106592025-04-30T06:43:19.273ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106602025-04-30T06:43:19.273ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106612025-04-30T06:43:19.274ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106622025-04-30T06:43:19.308ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
106632025-04-30T06:43:19.320ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
106642025-04-30T06:43:19.320ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
106652025-04-30T06:43:19.320ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 509.218434ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
106662025-04-30T06:43:19.321ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
106672025-04-30T06:43:19.321ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106682025-04-30T06:43:19.321ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106692025-04-30T06:43:19.321ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106702025-04-30T06:43:19.321ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106712025-04-30T06:43:19.322ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106722025-04-30T06:43:19.324ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106732025-04-30T06:43:19.330ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106742025-04-30T06:43:19.330ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106752025-04-30T06:43:19.330ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106762025-04-30T06:43:19.330ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106772025-04-30T06:43:19.344ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
106782025-04-30T06:43:19.344ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
106792025-04-30T06:43:19.344ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 00 e3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58500 req_id = 4e0e27a5-6c48-410e-946f-778bcbeaf2f1
106802025-04-30T06:43:19.344ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 227,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58500 req_id = 4e0e27a5-6c48-410e-946f-778bcbeaf2f1
106812025-04-30T06:43:19.345ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
106822025-04-30T06:43:19.345ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106832025-04-30T06:43:19.345ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58500 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4e0e27a5-6c48-410e-946f-778bcbeaf2f1
106842025-04-30T06:43:19.346ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4264, tv_nsec: 144899328 } })
106852025-04-30T06:43:19.346ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
106862025-04-30T06:43:19.352ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 10 f2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64078 req_id = b37cf564-d745-488b-a65d-caa1780e7657
106872025-04-30T06:43:19.352ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4338,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64078 req_id = b37cf564-d745-488b-a65d-caa1780e7657
106882025-04-30T06:43:19.352ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106892025-04-30T06:43:19.353ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64078 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b37cf564-d745-488b-a65d-caa1780e7657
106902025-04-30T06:43:19.409ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
106912025-04-30T06:43:19.409ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106922025-04-30T06:43:19.410ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106932025-04-30T06:43:19.410ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106942025-04-30T06:43:19.410ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
106952025-04-30T06:43:19.414ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106962025-04-30T06:43:19.418ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ab 84 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61283 req_id = c05383a4-ff27-4cd5-a235-d96d481574f9
106972025-04-30T06:43:19.418ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43908,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61283 req_id = c05383a4-ff27-4cd5-a235-d96d481574f9
106982025-04-30T06:43:19.418ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
106992025-04-30T06:43:19.419ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43908, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61283 req_id = c05383a4-ff27-4cd5-a235-d96d481574f9
107002025-04-30T06:43:19.419ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
107012025-04-30T06:43:19.419ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
107022025-04-30T06:43:19.420ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
107032025-04-30T06:43:19.421ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107042025-04-30T06:43:19.458ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107052025-04-30T06:43:19.463ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107062025-04-30T06:43:19.471ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107072025-04-30T06:43:19.478ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107082025-04-30T06:43:19.496ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4140e76-1860-4d50-826c-8667a8d4bbf4", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107092025-04-30T06:43:19.496ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
107102025-04-30T06:43:19.497ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107112025-04-30T06:43:19.497ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107122025-04-30T06:43:19.498ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107132025-04-30T06:43:19.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "149c3b65-5a92-4a6e-8edb-50632b476c01", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107142025-04-30T06:43:19.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
107152025-04-30T06:43:19.504ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59db3adb-bed1-4311-b252-7ca727842691", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107162025-04-30T06:43:19.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
107172025-04-30T06:43:19.508ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcd39cfd-7d7c-4363-a3ba-71b1d8f17c87", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107182025-04-30T06:43:19.508ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
107192025-04-30T06:43:19.512ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd2d736-1005-4327-87d2-429fdce09c09", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107202025-04-30T06:43:19.512ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
107212025-04-30T06:43:19.516ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc822f40-a930-4ef1-a5c8-7ae70cf03cd7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107222025-04-30T06:43:19.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
107232025-04-30T06:43:19.521ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87f58f22-f7f5-43af-818a-6a5b2bf86010", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107242025-04-30T06:43:19.521ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
107252025-04-30T06:43:19.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb0b99a7-c813-4f60-b754-b322be1c1321", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107262025-04-30T06:43:19.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
107272025-04-30T06:43:19.525ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
107282025-04-30T06:43:19.527ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03cb4568-934a-4998-851c-309c9582a93f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107292025-04-30T06:43:19.527ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
107302025-04-30T06:43:19.528ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f186688-8e9a-420f-a065-1483a0c3d88d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107312025-04-30T06:43:19.528ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
107322025-04-30T06:43:19.529ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44e30826-90b5-4fb3-b3b9-add7feb71111", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107332025-04-30T06:43:19.529ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
107342025-04-30T06:43:19.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41071f43-9ed4-4d53-a8d9-1709874fe29b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107352025-04-30T06:43:19.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
107362025-04-30T06:43:19.531ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5e423ce-760d-4efd-903b-9e22c0c0cdbf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107372025-04-30T06:43:19.532ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
107382025-04-30T06:43:19.534ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e6f4c1a-caf0-4f3a-bf3e-9bc4ec4457e5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107392025-04-30T06:43:19.534ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
107402025-04-30T06:43:19.540ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ee7867f-b823-47d6-b0ed-eaa93deb83af", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107412025-04-30T06:43:19.541ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
107422025-04-30T06:43:19.543ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "460730fb-75e6-47ff-a309-bab43014726b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107432025-04-30T06:43:19.543ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
107442025-04-30T06:43:19.549ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e57d4c9c-7da4-4420-9fe4-6f4e94ef3f28", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107452025-04-30T06:43:19.549ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
107462025-04-30T06:43:19.551ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c0b1a54-308f-47b2-a3e1-d83d4d791ed7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107472025-04-30T06:43:19.551ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
107482025-04-30T06:43:19.554ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fcc9910-0750-4abf-a8ea-c5bca251676f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107492025-04-30T06:43:19.554ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
107502025-04-30T06:43:19.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54ba9d9e-5d76-46d5-ba44-e68730996722", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107512025-04-30T06:43:19.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
107522025-04-30T06:43:19.558ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d806c2a0-49b8-4d13-a31c-26300cd43bbf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107532025-04-30T06:43:19.558ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
107542025-04-30T06:43:19.560ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30e555aa-8c8b-467b-8994-52da444398c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107552025-04-30T06:43:19.561ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
107562025-04-30T06:43:19.561ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d3e9511-5537-4929-85dd-c6c02e5ebeae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107572025-04-30T06:43:19.562ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
107582025-04-30T06:43:19.563ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce02f4ab-643e-4650-8940-f324e33761fc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107592025-04-30T06:43:19.564ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
107602025-04-30T06:43:19.569ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf70d5f8-c897-449f-a997-abfd90e9fd35", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107612025-04-30T06:43:19.570ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
107622025-04-30T06:43:19.576ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2306564-b747-4638-80ba-8ec714ca1730", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
107632025-04-30T06:43:19.576ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
107642025-04-30T06:43:19.577ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107652025-04-30T06:43:19.577ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107662025-04-30T06:43:19.577ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107672025-04-30T06:43:19.577ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107682025-04-30T06:43:19.579ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107692025-04-30T06:43:19.583ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107702025-04-30T06:43:19.600ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107712025-04-30T06:43:19.600ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107722025-04-30T06:43:19.600ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107732025-04-30T06:43:19.604ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107742025-04-30T06:43:19.607ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
107752025-04-30T06:43:19.607ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107762025-04-30T06:43:19.608ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3bf52fd7-41ad-41dc-b752-891650028737 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107772025-04-30T06:43:19.608ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3bf52fd7-41ad-41dc-b752-891650028737 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107782025-04-30T06:43:19.609ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3bf52fd7-41ad-41dc-b752-891650028737 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107792025-04-30T06:43:19.610ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3bf52fd7-41ad-41dc-b752-891650028737 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107802025-04-30T06:43:19.612ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107812025-04-30T06:43:19.612ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107822025-04-30T06:43:19.613ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107832025-04-30T06:43:19.615ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 7306 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 3bf52fd7-41ad-41dc-b752-891650028737 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107842025-04-30T06:43:19.616ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bf52fd7-41ad-41dc-b752-891650028737", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
107852025-04-30T06:43:19.617ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
107862025-04-30T06:43:19.618ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = dfd1a670-bfa4-42a5-8331-d217097d4823 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
107872025-04-30T06:43:19.618ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = dfd1a670-bfa4-42a5-8331-d217097d4823 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
107882025-04-30T06:43:19.619ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = dfd1a670-bfa4-42a5-8331-d217097d4823 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
107892025-04-30T06:43:19.619ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = dfd1a670-bfa4-42a5-8331-d217097d4823 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
107902025-04-30T06:43:19.620ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "660098ba-9d04-4101-a2a9-84ea98ac8895", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
107912025-04-30T06:43:19.620ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
107922025-04-30T06:43:19.623ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 5089 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = dfd1a670-bfa4-42a5-8331-d217097d4823 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
107932025-04-30T06:43:19.624ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfd1a670-bfa4-42a5-8331-d217097d4823", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
107942025-04-30T06:43:19.624ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
107952025-04-30T06:43:19.624ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107962025-04-30T06:43:19.624ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
107972025-04-30T06:43:19.627ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9770d38b-e1fc-434d-a373-fe3cca6be990", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
107982025-04-30T06:43:19.628ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
107992025-04-30T06:43:19.630ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2f7c830-fd79-4eb6-8384-b84d68a93e4b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108002025-04-30T06:43:19.631ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
108012025-04-30T06:43:19.633ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "79cf2425-ca39-4905-b997-29f8277fcb14", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108022025-04-30T06:43:19.634ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
108032025-04-30T06:43:19.635ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21564739-cbd4-4514-b6e9-e8122306b1ac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108042025-04-30T06:43:19.636ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
108052025-04-30T06:43:19.637ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59fb90fa-591a-4b01-b385-8a125f7a5e54", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108062025-04-30T06:43:19.638ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
108072025-04-30T06:43:19.639ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76adc5c7-f962-4f10-a9a9-8137289ead95", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108082025-04-30T06:43:19.639ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
108092025-04-30T06:43:19.642ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d394718-75a9-4a18-9705-ce385b748bb2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108102025-04-30T06:43:19.646ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
108112025-04-30T06:43:19.666ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d70fa9-eeef-4f62-948b-6ecfafc5a713", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108122025-04-30T06:43:19.666ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
108132025-04-30T06:43:19.667ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
108142025-04-30T06:43:19.674ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "080177eb-2858-429c-8cbc-fa50831008b4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108152025-04-30T06:43:19.675ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
108162025-04-30T06:43:19.678ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1769ec48-2b32-4aef-a5c7-f431a913a43c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108172025-04-30T06:43:19.678ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
108182025-04-30T06:43:19.682ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
108192025-04-30T06:43:19.682ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
108202025-04-30T06:43:19.682ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
108212025-04-30T06:43:19.685ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
108222025-04-30T06:43:19.761ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108232025-04-30T06:43:19.761ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108242025-04-30T06:43:19.761ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108252025-04-30T06:43:19.762ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
108262025-04-30T06:43:19.762ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108272025-04-30T06:43:19.763ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
108282025-04-30T06:43:19.763ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
108292025-04-30T06:43:19.764ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108302025-04-30T06:43:19.765ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108312025-04-30T06:43:19.775ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108322025-04-30T06:43:19.789ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108332025-04-30T06:43:19.790ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108342025-04-30T06:43:19.790ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
108352025-04-30T06:43:19.791ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108362025-04-30T06:43:19.791ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
108372025-04-30T06:43:19.791ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
108382025-04-30T06:43:19.791ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108392025-04-30T06:43:19.792ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108402025-04-30T06:43:19.792ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108412025-04-30T06:43:19.793ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108422025-04-30T06:43:19.793ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0389c7ab-3aac-41a3-a7e8-090a74fa0b98", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108432025-04-30T06:43:19.793ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
108442025-04-30T06:43:19.808ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
108452025-04-30T06:43:19.808ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108462025-04-30T06:43:19.809ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
108472025-04-30T06:43:19.809ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108482025-04-30T06:43:19.813ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa660269-f91d-43b1-bda8-02e4819cd94f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108492025-04-30T06:43:19.814ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
108502025-04-30T06:43:19.817ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1664e657-9eed-4544-b6ad-da0a8d58c235", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
108512025-04-30T06:43:19.818ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
108522025-04-30T06:43:19.836ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
108532025-04-30T06:43:19.836ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
108542025-04-30T06:43:19.839ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
108552025-04-30T06:43:19.839ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
108562025-04-30T06:43:19.839ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
108572025-04-30T06:43:19.839ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
108582025-04-30T06:43:19.839ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
108592025-04-30T06:43:19.839ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
108602025-04-30T06:43:19.842ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
108612025-04-30T06:43:19.843ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
108622025-04-30T06:43:19.845ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
108632025-04-30T06:43:19.845ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 56.322\u{b5}s iteration = 3 status = {"error":"task disabled"}
108642025-04-30T06:43:19.845ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
108652025-04-30T06:43:19.845ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
108662025-04-30T06:43:19.845ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108672025-04-30T06:43:19.845ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
108682025-04-30T06:43:19.845ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
108692025-04-30T06:43:19.848ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
108702025-04-30T06:43:19.848ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
108712025-04-30T06:43:19.848ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
108722025-04-30T06:43:19.848ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108732025-04-30T06:43:19.848ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
108742025-04-30T06:43:19.848ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108752025-04-30T06:43:19.849ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108762025-04-30T06:43:19.849ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
108772025-04-30T06:43:19.849ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
108782025-04-30T06:43:19.849ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
108792025-04-30T06:43:19.849ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
108802025-04-30T06:43:19.851ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
108812025-04-30T06:43:19.851ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
108822025-04-30T06:43:19.851ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108832025-04-30T06:43:19.851ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
108842025-04-30T06:43:19.851ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
108852025-04-30T06:43:19.851ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
108862025-04-30T06:43:19.851ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
108872025-04-30T06:43:19.851ZINFOtest_snapshot_expunge (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
108882025-04-30T06:43:19.851ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = instance_updater elapsed = 61.272\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
108892025-04-30T06:43:19.851ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
108902025-04-30T06:43:19.851ZINFOtest_snapshot_expunge (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
108912025-04-30T06:43:19.851ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
108922025-04-30T06:43:19.851ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108932025-04-30T06:43:19.851ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
108942025-04-30T06:43:19.851ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
108952025-04-30T06:43:19.852ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
108962025-04-30T06:43:19.852ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ff 51 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .Q..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:57772 req_id = 45d39346-370a-40bb-b503-257abdb39b34
108972025-04-30T06:43:19.852ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65361,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57772 req_id = 45d39346-370a-40bb-b503-257abdb39b34
108982025-04-30T06:43:19.852ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
108992025-04-30T06:43:19.852ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109002025-04-30T06:43:19.852ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57772 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 45d39346-370a-40bb-b503-257abdb39b34
109012025-04-30T06:43:19.853ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
109022025-04-30T06:43:19.856ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109032025-04-30T06:43:19.863ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
109042025-04-30T06:43:19.863ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
109052025-04-30T06:43:19.863ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
109062025-04-30T06:43:19.863ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
109072025-04-30T06:43:19.867ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109082025-04-30T06:43:19.879ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
109092025-04-30T06:43:19.880ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109102025-04-30T06:43:19.880ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109112025-04-30T06:43:19.883ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109122025-04-30T06:43:19.883ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 41.03539ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
109132025-04-30T06:43:19.887ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4264, tv_nsec: 686240766 } })
109142025-04-30T06:43:19.887ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
109152025-04-30T06:43:19.889ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68d2a232-1913-4f5f-9320-2f27ef218937", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
109162025-04-30T06:43:19.889ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
109172025-04-30T06:43:19.890ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109182025-04-30T06:43:19.912ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
109192025-04-30T06:43:19.912ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
109202025-04-30T06:43:19.912ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
109212025-04-30T06:43:19.912ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
109222025-04-30T06:43:19.917ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109232025-04-30T06:43:19.920ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109242025-04-30T06:43:19.920ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
109252025-04-30T06:43:19.920ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109262025-04-30T06:43:19.920ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109272025-04-30T06:43:19.920ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109282025-04-30T06:43:19.936ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109292025-04-30T06:43:19.936ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
109302025-04-30T06:43:19.937ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109312025-04-30T06:43:19.937ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109322025-04-30T06:43:19.938ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109332025-04-30T06:43:19.938ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109342025-04-30T06:43:19.938ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109352025-04-30T06:43:19.939ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109362025-04-30T06:43:19.941ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 94 d9 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53491 req_id = d930c08c-b7d1-4cf9-aa4d-b29116102594
109372025-04-30T06:43:19.941ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38105,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53491 req_id = d930c08c-b7d1-4cf9-aa4d-b29116102594
109382025-04-30T06:43:19.942ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
109392025-04-30T06:43:19.942ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109402025-04-30T06:43:19.942ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53491 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d930c08c-b7d1-4cf9-aa4d-b29116102594
109412025-04-30T06:43:19.944ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109422025-04-30T06:43:19.949ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109432025-04-30T06:43:19.949ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109442025-04-30T06:43:19.949ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109452025-04-30T06:43:19.949ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109462025-04-30T06:43:19.960ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
109472025-04-30T06:43:19.960ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109482025-04-30T06:43:19.960ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109492025-04-30T06:43:19.960ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109502025-04-30T06:43:19.961ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109512025-04-30T06:43:19.962ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4264, tv_nsec: 761886455 } })
109522025-04-30T06:43:19.963ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
109532025-04-30T06:43:19.963ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 09 1a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55704 req_id = e023e105-05a0-487f-b73b-3e3a07e7d311
109542025-04-30T06:43:19.963ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2330,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55704 req_id = e023e105-05a0-487f-b73b-3e3a07e7d311
109552025-04-30T06:43:19.964ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109562025-04-30T06:43:19.964ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55704 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e023e105-05a0-487f-b73b-3e3a07e7d311
109572025-04-30T06:43:19.964ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 9d f5 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64900 req_id = 16919d1d-633f-497d-a57b-9a225fd779fd
109582025-04-30T06:43:19.964ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40437,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64900 req_id = 16919d1d-633f-497d-a57b-9a225fd779fd
109592025-04-30T06:43:19.964ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
109602025-04-30T06:43:19.964ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109612025-04-30T06:43:19.964ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64900 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 16919d1d-633f-497d-a57b-9a225fd779fd
109622025-04-30T06:43:20.062ZINFOtest_snapshot_expunge (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
109632025-04-30T06:43:20.063ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:51119/vpc-routes
109642025-04-30T06:43:20.063ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109652025-04-30T06:43:20.073ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
109662025-04-30T06:43:20.073ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
109672025-04-30T06:43:20.073ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109682025-04-30T06:43:20.073ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109692025-04-30T06:43:20.073ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109702025-04-30T06:43:20.073ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109712025-04-30T06:43:20.075ZINFOtest_snapshot_expunge (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
109722025-04-30T06:43:20.075ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 224.482226ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
109732025-04-30T06:43:20.078ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:56373 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
109742025-04-30T06:43:20.079ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
109752025-04-30T06:43:20.149ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
109762025-04-30T06:43:20.149ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
109772025-04-30T06:43:20.149ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
109782025-04-30T06:43:20.149ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
109792025-04-30T06:43:20.150ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
109802025-04-30T06:43:20.151ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4264, tv_nsec: 950211510 } })
109812025-04-30T06:43:20.151ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
109822025-04-30T06:43:20.153ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109832025-04-30T06:43:20.157ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
109842025-04-30T06:43:20.158ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109852025-04-30T06:43:20.158ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109862025-04-30T06:43:20.158ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109872025-04-30T06:43:20.162ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:56373 req_id = 3404a1f4-6bbe-4aef-a858-9aefc678eef5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
109882025-04-30T06:43:20.164ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 64 b6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e d...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56304 req_id = 15a21d74-5fde-4dc2-93c1-9f26c2dffa52
109892025-04-30T06:43:20.164ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25782,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56304 req_id = 15a21d74-5fde-4dc2-93c1-9f26c2dffa52
109902025-04-30T06:43:20.165ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109912025-04-30T06:43:20.170ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56304 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 15a21d74-5fde-4dc2-93c1-9f26c2dffa52
109922025-04-30T06:43:20.170ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac4c4bd0-5cad-4558-ae1d-fe26e7fe1217", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
109932025-04-30T06:43:20.171ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
109942025-04-30T06:43:20.172ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 10291 local_addr = [::1]:51119 method = GET remote_addr = [::1]:56373 req_id = 3404a1f4-6bbe-4aef-a858-9aefc678eef5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
109952025-04-30T06:43:20.184ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
109962025-04-30T06:43:20.242ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
109972025-04-30T06:43:20.242ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
109982025-04-30T06:43:20.242ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
109992025-04-30T06:43:20.242ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
110002025-04-30T06:43:20.244ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
110012025-04-30T06:43:20.244ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2f 26 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e /&..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62616 req_id = b320a03e-fd1e-4722-8570-c8ceac14f1b5
110022025-04-30T06:43:20.244ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12070,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62616 req_id = b320a03e-fd1e-4722-8570-c8ceac14f1b5
110032025-04-30T06:43:20.244ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110042025-04-30T06:43:20.245ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62616 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b320a03e-fd1e-4722-8570-c8ceac14f1b5
110052025-04-30T06:43:20.245ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:51119/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3404a1f4-6bbe-4aef-a858-9aefc678eef5", "content-length": "2", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
110062025-04-30T06:43:20.245ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110072025-04-30T06:43:20.245ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110082025-04-30T06:43:20.246ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110092025-04-30T06:43:20.297ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:51119/v2p
110102025-04-30T06:43:20.300ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
110112025-04-30T06:43:20.303ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:38039 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
110122025-04-30T06:43:20.303ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
110132025-04-30T06:43:20.352ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
110142025-04-30T06:43:20.352ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
110152025-04-30T06:43:20.355ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110162025-04-30T06:43:20.355ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110172025-04-30T06:43:20.355ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110182025-04-30T06:43:20.355ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
110192025-04-30T06:43:20.356ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110202025-04-30T06:43:20.357ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
110212025-04-30T06:43:20.357ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
110222025-04-30T06:43:20.358ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
110232025-04-30T06:43:20.358ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:38039 req_id = f5867df7-06f9-476f-88ce-996f20725327 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
110242025-04-30T06:43:20.363ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 4452 local_addr = [::1]:51119 method = GET remote_addr = [::1]:38039 req_id = f5867df7-06f9-476f-88ce-996f20725327 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
110252025-04-30T06:43:20.366ZINFOtest_snapshot_expunge (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
110262025-04-30T06:43:20.366ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:51119/eip-gateways
110272025-04-30T06:43:20.368ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = PUT remote_addr = [::1]:56373 req_id = 6421f710-137a-4575-8b53-9d523806c660 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
110282025-04-30T06:43:20.368ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:51119/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5867df7-06f9-476f-88ce-996f20725327", "content-length": "2", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
110292025-04-30T06:43:20.368ZINFOtest_snapshot_expunge (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
110302025-04-30T06:43:20.369ZINFOtest_snapshot_expunge (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
110312025-04-30T06:43:20.369ZINFOtest_snapshot_expunge (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
110322025-04-30T06:43:20.370ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 521.549583ms iteration = 2 status = {}
110332025-04-30T06:43:20.371ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 2754 local_addr = [::1]:51119 method = PUT remote_addr = [::1]:56373 req_id = 6421f710-137a-4575-8b53-9d523806c660 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
110342025-04-30T06:43:20.371ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110352025-04-30T06:43:20.378ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
110362025-04-30T06:43:20.378ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110372025-04-30T06:43:20.379ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
110382025-04-30T06:43:20.379ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110392025-04-30T06:43:20.383ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:51119/eip-gateways", status: 204, headers: {"x-request-id": "6421f710-137a-4575-8b53-9d523806c660", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
110402025-04-30T06:43:20.386ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 534.91797ms iteration = 2 status = {}
110412025-04-30T06:43:20.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbe8f4ad-7011-4d54-8287-fea7e9542c44", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
110422025-04-30T06:43:20.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
110432025-04-30T06:43:20.391ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110442025-04-30T06:43:20.391ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
110452025-04-30T06:43:20.392ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110462025-04-30T06:43:20.414ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87d15411-ad9b-4622-bdb4-1d605354baa6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
110472025-04-30T06:43:20.415ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
110482025-04-30T06:43:20.422ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "414b7a1b-c530-485a-a5c1-b5d4f97f52fa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
110492025-04-30T06:43:20.422ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
110502025-04-30T06:43:20.422ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110512025-04-30T06:43:20.422ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110522025-04-30T06:43:20.422ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110532025-04-30T06:43:20.423ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110542025-04-30T06:43:20.424ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110552025-04-30T06:43:20.424ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110562025-04-30T06:43:20.424ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110572025-04-30T06:43:20.429ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110582025-04-30T06:43:20.434ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
110592025-04-30T06:43:20.434ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
110602025-04-30T06:43:20.434ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110612025-04-30T06:43:20.434ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110622025-04-30T06:43:20.434ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110632025-04-30T06:43:20.434ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110642025-04-30T06:43:20.435ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
110652025-04-30T06:43:20.435ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110662025-04-30T06:43:20.435ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110672025-04-30T06:43:20.435ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110682025-04-30T06:43:20.435ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110692025-04-30T06:43:20.438ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110702025-04-30T06:43:20.442ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
110712025-04-30T06:43:20.442ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110722025-04-30T06:43:20.442ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110732025-04-30T06:43:20.442ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110742025-04-30T06:43:20.442ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110752025-04-30T06:43:20.455ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
110762025-04-30T06:43:20.455ZERROtest_snapshot_expunge (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
110772025-04-30T06:43:20.455ZERROtest_snapshot_expunge (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
110782025-04-30T06:43:20.456ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 610.237669ms iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
110792025-04-30T06:43:20.584ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
110802025-04-30T06:43:20.584ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
110812025-04-30T06:43:20.585ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:45584 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
110822025-04-30T06:43:20.591ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:45584 req_id = e83f6b27-4bbb-4a19-90df-075af61c54d7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
110832025-04-30T06:43:20.593ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 2281 local_addr = [::1]:51119 method = GET remote_addr = [::1]:45584 req_id = e83f6b27-4bbb-4a19-90df-075af61c54d7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
110842025-04-30T06:43:20.594ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e83f6b27-4bbb-4a19-90df-075af61c54d7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
110852025-04-30T06:43:20.600ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
110862025-04-30T06:43:20.600ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
110872025-04-30T06:43:20.600ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
110882025-04-30T06:43:20.606ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
110892025-04-30T06:43:20.606ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110902025-04-30T06:43:20.606ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110912025-04-30T06:43:20.606ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110922025-04-30T06:43:20.607ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110932025-04-30T06:43:20.612ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110942025-04-30T06:43:20.619ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
110952025-04-30T06:43:20.620ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110962025-04-30T06:43:20.620ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110972025-04-30T06:43:20.626ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110982025-04-30T06:43:20.638ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
110992025-04-30T06:43:20.666ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
111002025-04-30T06:43:20.666ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
111012025-04-30T06:43:20.667ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.345808857s iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
111022025-04-30T06:43:20.667ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
111032025-04-30T06:43:20.667ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
111042025-04-30T06:43:20.667ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111052025-04-30T06:43:20.667ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111062025-04-30T06:43:20.667ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111072025-04-30T06:43:20.668ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111082025-04-30T06:43:20.670ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111092025-04-30T06:43:20.680ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
111102025-04-30T06:43:20.680ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111112025-04-30T06:43:20.680ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111122025-04-30T06:43:20.680ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111132025-04-30T06:43:20.687ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
111142025-04-30T06:43:20.687ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
111152025-04-30T06:43:20.689ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111162025-04-30T06:43:20.689ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111172025-04-30T06:43:20.689ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111182025-04-30T06:43:20.689ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
111192025-04-30T06:43:20.691ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111202025-04-30T06:43:20.691ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111212025-04-30T06:43:20.691ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111222025-04-30T06:43:20.691ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
111232025-04-30T06:43:20.693ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
111242025-04-30T06:43:20.693ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: db d4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50507 req_id = 5018dac1-c8c6-49a9-b032-668b1fec6b5e
111252025-04-30T06:43:20.694ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56276,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50507 req_id = 5018dac1-c8c6-49a9-b032-668b1fec6b5e
111262025-04-30T06:43:20.694ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
111272025-04-30T06:43:20.694ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111282025-04-30T06:43:20.695ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50507 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5018dac1-c8c6-49a9-b032-668b1fec6b5e
111292025-04-30T06:43:20.696ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4265, tv_nsec: 495757495 } })
111302025-04-30T06:43:20.696ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
111312025-04-30T06:43:20.697ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111322025-04-30T06:43:20.704ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111332025-04-30T06:43:20.705ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111342025-04-30T06:43:20.705ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
111352025-04-30T06:43:20.708ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111362025-04-30T06:43:20.708ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
111372025-04-30T06:43:20.708ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
111382025-04-30T06:43:20.709ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: da 28 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .(..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62597 req_id = cd11e336-455e-4621-8be5-57dd25dff3d8
111392025-04-30T06:43:20.709ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55848,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62597 req_id = cd11e336-455e-4621-8be5-57dd25dff3d8
111402025-04-30T06:43:20.710ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111412025-04-30T06:43:20.710ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62597 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cd11e336-455e-4621-8be5-57dd25dff3d8
111422025-04-30T06:43:20.792ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
111432025-04-30T06:43:20.793ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111442025-04-30T06:43:20.793ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
111452025-04-30T06:43:20.797ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111462025-04-30T06:43:20.797ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
111472025-04-30T06:43:20.797ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111482025-04-30T06:43:20.798ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111492025-04-30T06:43:20.798ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111502025-04-30T06:43:20.798ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
111512025-04-30T06:43:20.799ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 63 fd 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c c............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51943 req_id = 89337f5f-22f4-4d41-8f25-6b11d69e804f
111522025-04-30T06:43:20.799ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25597,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51943 req_id = 89337f5f-22f4-4d41-8f25-6b11d69e804f
111532025-04-30T06:43:20.800ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
111542025-04-30T06:43:20.800ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25597, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51943 req_id = 89337f5f-22f4-4d41-8f25-6b11d69e804f
111552025-04-30T06:43:20.802ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
111562025-04-30T06:43:20.802ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
111572025-04-30T06:43:20.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
111582025-04-30T06:43:20.802ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111592025-04-30T06:43:20.803ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111602025-04-30T06:43:20.803ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111612025-04-30T06:43:20.803ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111622025-04-30T06:43:20.808ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "eba082d8-492d-4581-9c57-519bcd09a086", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111632025-04-30T06:43:20.808ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
111642025-04-30T06:43:20.808ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3269efa-b8c4-4e49-a9f0-890904e4ed12", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111652025-04-30T06:43:20.808ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
111662025-04-30T06:43:20.813ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94169b0f-a6dd-4c9c-862c-53919a798e4c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111672025-04-30T06:43:20.813ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
111682025-04-30T06:43:20.817ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cafbcc20-a703-405f-8f74-561c9bb3d5f2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111692025-04-30T06:43:20.817ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
111702025-04-30T06:43:20.829ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a191ec2-d001-44b9-9a7d-aa2191295d73", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111712025-04-30T06:43:20.829ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
111722025-04-30T06:43:20.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffba6f8-7250-4c9d-a6b3-22e6907bcc0e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111732025-04-30T06:43:20.833ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
111742025-04-30T06:43:20.837ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28c6ffa2-ae2f-4510-8d7e-289e31e5ec9b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111752025-04-30T06:43:20.837ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
111762025-04-30T06:43:20.846ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29c0a63a-fbbb-4ba2-abb5-4acc2ad4fc2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111772025-04-30T06:43:20.850ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
111782025-04-30T06:43:20.856ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a0613c5-b2af-443d-8f4b-f0237c166a52", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111792025-04-30T06:43:20.856ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
111802025-04-30T06:43:20.862ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
111812025-04-30T06:43:20.864ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "551f78d1-ef37-46ca-8545-021043460caa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111822025-04-30T06:43:20.864ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
111832025-04-30T06:43:20.866ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cab3200-df9a-4a98-b48b-04b8715cc1bc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111842025-04-30T06:43:20.866ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
111852025-04-30T06:43:20.867ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8adb16fd-484b-41b9-bfc0-52062337d600", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111862025-04-30T06:43:20.872ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
111872025-04-30T06:43:20.876ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6bcb959-c2b7-46e7-a3d4-a16257290267", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111882025-04-30T06:43:20.877ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
111892025-04-30T06:43:20.879ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a701c6a0-b897-4bb6-aec2-c2792850b56e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111902025-04-30T06:43:20.879ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
111912025-04-30T06:43:20.885ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f03e9a9-1a99-4d79-8a90-7d7817281bf9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111922025-04-30T06:43:20.885ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
111932025-04-30T06:43:20.889ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a50aa19-f0a9-48f8-a3cc-ba5cf7f176a6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111942025-04-30T06:43:20.889ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
111952025-04-30T06:43:20.890ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ae1cdfd-5771-4d6d-b98e-fcca268533c1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111962025-04-30T06:43:20.890ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
111972025-04-30T06:43:20.893ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60a948ce-90fa-4594-a899-32f2e852c2b8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
111982025-04-30T06:43:20.894ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
111992025-04-30T06:43:20.898ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b250d136-991b-4786-9e8a-62ecd4e628fc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112002025-04-30T06:43:20.898ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
112012025-04-30T06:43:20.900ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff84f7c-612f-427c-b024-fd61b8772e73", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112022025-04-30T06:43:20.901ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
112032025-04-30T06:43:20.911ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c5cf076-38a3-4929-b6e0-004b533be892", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112042025-04-30T06:43:20.911ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
112052025-04-30T06:43:20.912ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cae6b10-2735-4331-a0df-708faca601ea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112062025-04-30T06:43:20.913ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
112072025-04-30T06:43:20.913ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112082025-04-30T06:43:20.913ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112092025-04-30T06:43:20.913ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112102025-04-30T06:43:20.913ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112112025-04-30T06:43:20.914ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112122025-04-30T06:43:20.918ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ca825f0-22ef-49b5-9159-a741cbed241d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112132025-04-30T06:43:20.918ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
112142025-04-30T06:43:20.918ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112152025-04-30T06:43:20.926ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112162025-04-30T06:43:20.926ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112172025-04-30T06:43:20.926ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112182025-04-30T06:43:20.927ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112192025-04-30T06:43:20.929ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3eac485-e437-4021-a43e-25b23fda25fb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112202025-04-30T06:43:20.930ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
112212025-04-30T06:43:20.934ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112222025-04-30T06:43:20.934ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112232025-04-30T06:43:20.937ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112242025-04-30T06:43:20.938ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca7a456-6dc2-4ee9-9d59-7d7189b4d7a7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112252025-04-30T06:43:20.938ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
112262025-04-30T06:43:20.942ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d98d71a3-ca1f-4ccd-8a20-3cab6482d6c0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112272025-04-30T06:43:20.942ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
112282025-04-30T06:43:20.945ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9166758d-751b-4891-afb1-7f266d853928", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112292025-04-30T06:43:20.945ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
112302025-04-30T06:43:20.950ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cbd7a22-9985-40e2-af48-713b76418025", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112312025-04-30T06:43:20.950ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
112322025-04-30T06:43:20.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5045bb91-b729-4d1c-84ba-910d85309bc8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112332025-04-30T06:43:20.954ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
112342025-04-30T06:43:20.961ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fe2b82d-6ba2-44df-87d0-b85d8d5e4599", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112352025-04-30T06:43:20.961ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
112362025-04-30T06:43:20.962ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3b69644-9af0-4d4a-854b-a4e3a123ad6f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112372025-04-30T06:43:20.962ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
112382025-04-30T06:43:20.964ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e651a4f1-4561-4091-bd4a-b574da3a0631", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112392025-04-30T06:43:20.964ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
112402025-04-30T06:43:20.966ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1048f94e-ea51-46e9-a898-8cb039a826e1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112412025-04-30T06:43:20.966ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
112422025-04-30T06:43:20.967ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de714d75-9a0c-4fb5-9e8e-d16626c68908", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112432025-04-30T06:43:20.967ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
112442025-04-30T06:43:20.968ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85793daa-d026-4319-808c-928cbb1b9392", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112452025-04-30T06:43:20.969ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
112462025-04-30T06:43:20.971ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74b7071c-777b-4941-9370-da24d70c812c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112472025-04-30T06:43:20.971ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
112482025-04-30T06:43:20.973ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0b58ec6-6048-44f3-9b6a-0d16dfdc3928", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112492025-04-30T06:43:20.973ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
112502025-04-30T06:43:20.979ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cc75b30-ba2b-488a-a008-b5c4c174992b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112512025-04-30T06:43:20.984ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
112522025-04-30T06:43:21.006ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
112532025-04-30T06:43:21.037ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e05aad2e-50ec-4100-9104-f6bbd9bf8369", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112542025-04-30T06:43:21.037ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
112552025-04-30T06:43:21.047ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b12edbb5-84af-4342-8424-d1b40920fc3d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112562025-04-30T06:43:21.047ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
112572025-04-30T06:43:21.049ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "211d2b81-44f2-49b4-b497-ce7bfb13ea2b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112582025-04-30T06:43:21.050ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
112592025-04-30T06:43:21.053ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e2e0d40-3336-4aaa-b2c0-8c5deeaaabce", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112602025-04-30T06:43:21.053ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
112612025-04-30T06:43:21.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9df8aced-b170-4bd9-a4f8-ed788a45c6b4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112622025-04-30T06:43:21.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
112632025-04-30T06:43:21.056ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "676d4f33-b682-48d3-8a35-5f9fdb353314", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112642025-04-30T06:43:21.057ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
112652025-04-30T06:43:21.057ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112662025-04-30T06:43:21.057ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112672025-04-30T06:43:21.057ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112682025-04-30T06:43:21.057ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112692025-04-30T06:43:21.058ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112702025-04-30T06:43:21.059ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff7f003-8708-40b6-9d29-d6ab1b4e90b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
112712025-04-30T06:43:21.060ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112722025-04-30T06:43:21.060ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112732025-04-30T06:43:21.060ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112742025-04-30T06:43:21.060ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112752025-04-30T06:43:21.061ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112762025-04-30T06:43:21.077ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112772025-04-30T06:43:21.084ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112782025-04-30T06:43:21.084ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112792025-04-30T06:43:21.084ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112802025-04-30T06:43:21.084ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112812025-04-30T06:43:21.088ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112822025-04-30T06:43:21.093ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112832025-04-30T06:43:21.094ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112842025-04-30T06:43:21.094ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112852025-04-30T06:43:21.095ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112862025-04-30T06:43:21.095ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112872025-04-30T06:43:21.096ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112882025-04-30T06:43:21.099ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112892025-04-30T06:43:21.099ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112902025-04-30T06:43:21.100ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112912025-04-30T06:43:21.102ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112922025-04-30T06:43:21.114ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112932025-04-30T06:43:21.118ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112942025-04-30T06:43:21.118ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112952025-04-30T06:43:21.119ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112962025-04-30T06:43:21.197ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
112972025-04-30T06:43:21.197ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
112982025-04-30T06:43:21.198ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:46864 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
112992025-04-30T06:43:21.198ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:46864 req_id = 5c2732c7-e30c-4f6a-9d45-1cbd2cf27ed8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113002025-04-30T06:43:21.199ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 550 local_addr = [::1]:51119 method = GET remote_addr = [::1]:46864 req_id = 5c2732c7-e30c-4f6a-9d45-1cbd2cf27ed8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113012025-04-30T06:43:21.199ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c2732c7-e30c-4f6a-9d45-1cbd2cf27ed8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113022025-04-30T06:43:21.201ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113032025-04-30T06:43:21.202ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113042025-04-30T06:43:21.202ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
113052025-04-30T06:43:21.206ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113062025-04-30T06:43:21.206ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113072025-04-30T06:43:21.206ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113082025-04-30T06:43:21.206ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113092025-04-30T06:43:21.207ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113102025-04-30T06:43:21.211ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113112025-04-30T06:43:21.216ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
113122025-04-30T06:43:21.217ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113132025-04-30T06:43:21.217ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113142025-04-30T06:43:21.217ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113152025-04-30T06:43:21.241ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
113162025-04-30T06:43:21.257ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
113172025-04-30T06:43:21.258ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
113182025-04-30T06:43:21.260ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 591.658161ms iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
113192025-04-30T06:43:21.260ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
113202025-04-30T06:43:21.260ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113212025-04-30T06:43:21.260ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113222025-04-30T06:43:21.260ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113232025-04-30T06:43:21.260ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113242025-04-30T06:43:21.260ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113252025-04-30T06:43:21.264ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113262025-04-30T06:43:21.270ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
113272025-04-30T06:43:21.270ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113282025-04-30T06:43:21.270ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113292025-04-30T06:43:21.271ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113302025-04-30T06:43:21.275ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
113312025-04-30T06:43:21.275ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
113322025-04-30T06:43:21.276ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5a 09 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Z............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52006 req_id = 8e1ed3fa-a231-4247-b7c0-0dbe8ce70961
113332025-04-30T06:43:21.276ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23049,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52006 req_id = 8e1ed3fa-a231-4247-b7c0-0dbe8ce70961
113342025-04-30T06:43:21.277ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
113352025-04-30T06:43:21.277ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113362025-04-30T06:43:21.277ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52006 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8e1ed3fa-a231-4247-b7c0-0dbe8ce70961
113372025-04-30T06:43:21.277ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4266, tv_nsec: 76488133 } })
113382025-04-30T06:43:21.277ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
113392025-04-30T06:43:21.278ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e 3f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .?..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53402 req_id = b1e606c8-52b2-4026-abd3-80d1c802e92d
113402025-04-30T06:43:21.279ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40511,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53402 req_id = b1e606c8-52b2-4026-abd3-80d1c802e92d
113412025-04-30T06:43:21.280ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113422025-04-30T06:43:21.280ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53402 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b1e606c8-52b2-4026-abd3-80d1c802e92d
113432025-04-30T06:43:21.364ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
113442025-04-30T06:43:21.365ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113452025-04-30T06:43:21.365ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113462025-04-30T06:43:21.365ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113472025-04-30T06:43:21.365ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
113482025-04-30T06:43:21.366ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113492025-04-30T06:43:21.366ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b6 b4 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53162 req_id = 77e416c3-5299-4578-bb3e-121ef456e9d5
113502025-04-30T06:43:21.366ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46772,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53162 req_id = 77e416c3-5299-4578-bb3e-121ef456e9d5
113512025-04-30T06:43:21.366ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
113522025-04-30T06:43:21.366ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46772, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53162 req_id = 77e416c3-5299-4578-bb3e-121ef456e9d5
113532025-04-30T06:43:21.366ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
113542025-04-30T06:43:21.367ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
113552025-04-30T06:43:21.367ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
113562025-04-30T06:43:21.369ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113572025-04-30T06:43:21.378ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113582025-04-30T06:43:21.378ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113592025-04-30T06:43:21.378ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
113602025-04-30T06:43:21.379ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113612025-04-30T06:43:21.380ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb99b107-9823-44a6-906a-605a0015b2fc", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113622025-04-30T06:43:21.380ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
113632025-04-30T06:43:21.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cea626b-fec0-452a-a62c-39de585ecfb5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113642025-04-30T06:43:21.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
113652025-04-30T06:43:21.384ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113662025-04-30T06:43:21.384ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
113672025-04-30T06:43:21.384ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113682025-04-30T06:43:21.385ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73da117f-34b6-4747-be28-fbbc68e9ab90", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113692025-04-30T06:43:21.385ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
113702025-04-30T06:43:21.388ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c5510ce-226b-49b0-be93-a89089219579", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113712025-04-30T06:43:21.388ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
113722025-04-30T06:43:21.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f994e6d8-ecda-4955-9ff3-8f75aad01b43", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113732025-04-30T06:43:21.390ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
113742025-04-30T06:43:21.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a627d8f5-daeb-48ee-b966-2b75b994867c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113752025-04-30T06:43:21.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
113762025-04-30T06:43:21.395ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57e64a1b-6b9d-4786-91da-bf2f2eaaadce", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113772025-04-30T06:43:21.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
113782025-04-30T06:43:21.399ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e202ca-a4ed-449b-a1f3-14534c3b52d0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113792025-04-30T06:43:21.399ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
113802025-04-30T06:43:21.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80071003-1f30-46b7-b38c-af77472bff6f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113812025-04-30T06:43:21.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
113822025-04-30T06:43:21.403ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fa5365d-b49f-4268-aee1-6080f172836c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113832025-04-30T06:43:21.403ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
113842025-04-30T06:43:21.405ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e0f3fb0-54bb-4243-b28d-89df028b9a54", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113852025-04-30T06:43:21.405ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
113862025-04-30T06:43:21.406ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c90a7f0e-92a8-4fb2-a9df-1e60ac8b9910", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113872025-04-30T06:43:21.406ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
113882025-04-30T06:43:21.408ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a564651f-0705-49df-a430-09819a07c45c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113892025-04-30T06:43:21.408ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
113902025-04-30T06:43:21.412ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87a00331-47dd-4cc5-8368-b0493525776b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113912025-04-30T06:43:21.412ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
113922025-04-30T06:43:21.416ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18ba9419-c82b-444c-a39e-e2087e5c00a4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113932025-04-30T06:43:21.416ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
113942025-04-30T06:43:21.418ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a153ec7-7559-4137-bb90-4edba50c5e0d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113952025-04-30T06:43:21.418ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
113962025-04-30T06:43:21.425ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffda443a-1064-4269-a27b-b5dd6f35df4d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113972025-04-30T06:43:21.425ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
113982025-04-30T06:43:21.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "330b111f-6ae2-477c-a0db-1f577c17e6b6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
113992025-04-30T06:43:21.434ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
114002025-04-30T06:43:21.434ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
114012025-04-30T06:43:21.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f793b355-4403-416b-be7c-83cfe92ab5e6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114022025-04-30T06:43:21.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
114032025-04-30T06:43:21.440ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "289cef27-7bf5-4df4-a9d8-997fc83b57dc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114042025-04-30T06:43:21.440ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
114052025-04-30T06:43:21.442ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac56c5db-cadd-454f-9430-ca0f8f638cab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114062025-04-30T06:43:21.442ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
114072025-04-30T06:43:21.443ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "238cc969-d018-4a96-9664-210644bde7ee", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114082025-04-30T06:43:21.444ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
114092025-04-30T06:43:21.445ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd5839e4-e0c2-4423-8f37-ec2e161317ef", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114102025-04-30T06:43:21.445ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
114112025-04-30T06:43:21.448ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c620383b-c5e7-40df-bd89-c5204a19172e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114122025-04-30T06:43:21.450ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
114132025-04-30T06:43:21.453ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58f6f46e-0852-46c1-9c1b-93eabbccef6d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114142025-04-30T06:43:21.453ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
114152025-04-30T06:43:21.465ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d8a8957-f4ee-4ca2-a75b-25b95857e007", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114162025-04-30T06:43:21.465ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
114172025-04-30T06:43:21.468ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2696561-848c-4195-afaf-43681eebf380", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114182025-04-30T06:43:21.468ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
114192025-04-30T06:43:21.471ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b488c49e-24b0-4b38-8d06-feb9663be723", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114202025-04-30T06:43:21.471ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
114212025-04-30T06:43:21.483ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a7c3fdf-31d1-4dbc-af42-aaf5896bde9c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114222025-04-30T06:43:21.483ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
114232025-04-30T06:43:21.485ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114242025-04-30T06:43:21.485ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114252025-04-30T06:43:21.485ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114262025-04-30T06:43:21.485ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
114272025-04-30T06:43:21.486ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114282025-04-30T06:43:21.487ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccc44ca4-79d2-4dc7-924e-3dedfdbec5ba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114292025-04-30T06:43:21.487ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
114302025-04-30T06:43:21.489ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5a08b3f-8c80-47ad-9257-a31ec5decd12", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114312025-04-30T06:43:21.490ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
114322025-04-30T06:43:21.490ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114332025-04-30T06:43:21.498ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114342025-04-30T06:43:21.498ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114352025-04-30T06:43:21.498ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
114362025-04-30T06:43:21.499ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114372025-04-30T06:43:21.501ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9db2203-14b0-47fa-876a-bc4f63380738", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114382025-04-30T06:43:21.501ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
114392025-04-30T06:43:21.502ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114402025-04-30T06:43:21.502ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
114412025-04-30T06:43:21.503ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114422025-04-30T06:43:21.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96d724d2-44e9-49c6-a6f0-4c7cefd61240", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114432025-04-30T06:43:21.504ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
114442025-04-30T06:43:21.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1edc0c67-244f-4bec-8132-1b676c1173f2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114452025-04-30T06:43:21.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
114462025-04-30T06:43:21.507ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5af0ad3-f302-44c5-a16c-89969450dac8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114472025-04-30T06:43:21.508ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
114482025-04-30T06:43:21.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19531df4-bad7-44c2-88ee-63584e47bbf1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114492025-04-30T06:43:21.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
114502025-04-30T06:43:21.518ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b76e66d-feea-408f-b1eb-358f3f028088", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114512025-04-30T06:43:21.518ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
114522025-04-30T06:43:21.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b910cb5a-37ce-44a1-8654-175047927320", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114532025-04-30T06:43:21.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
114542025-04-30T06:43:21.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3feca0a4-66e8-4c38-8c58-7f39ed380f5e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114552025-04-30T06:43:21.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
114562025-04-30T06:43:21.542ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb5223b0-914c-40c9-abfc-a333c91ee1c7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114572025-04-30T06:43:21.542ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
114582025-04-30T06:43:21.548ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
114592025-04-30T06:43:21.549ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3f74f71-a5c4-4219-a59f-0257f8676a8c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114602025-04-30T06:43:21.549ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
114612025-04-30T06:43:21.552ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "412beeb5-8f56-4624-a171-3aad14b9757c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114622025-04-30T06:43:21.553ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
114632025-04-30T06:43:21.555ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04c2e7dc-320e-4aba-a4fd-2bd68cf58d36", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114642025-04-30T06:43:21.556ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
114652025-04-30T06:43:21.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c472f9c-da63-4c46-9370-76a576bc2088", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114662025-04-30T06:43:21.557ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
114672025-04-30T06:43:21.559ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76d21d57-4289-4d88-8eb3-893156464d17", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
114682025-04-30T06:43:21.559ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114692025-04-30T06:43:21.560ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114702025-04-30T06:43:21.560ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114712025-04-30T06:43:21.560ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114722025-04-30T06:43:21.561ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114732025-04-30T06:43:21.565ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114742025-04-30T06:43:21.571ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
114752025-04-30T06:43:21.571ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114762025-04-30T06:43:21.571ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114772025-04-30T06:43:21.572ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114782025-04-30T06:43:21.572ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114792025-04-30T06:43:21.573ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114802025-04-30T06:43:21.579ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114812025-04-30T06:43:21.586ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
114822025-04-30T06:43:21.586ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114832025-04-30T06:43:21.586ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114842025-04-30T06:43:21.587ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114852025-04-30T06:43:21.588ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
114862025-04-30T06:43:21.588ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114872025-04-30T06:43:21.589ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = f0a3926a-c401-41d4-91cd-78e4f1c3c04c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114882025-04-30T06:43:21.589ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = f0a3926a-c401-41d4-91cd-78e4f1c3c04c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114892025-04-30T06:43:21.589ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = f0a3926a-c401-41d4-91cd-78e4f1c3c04c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114902025-04-30T06:43:21.593ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = f0a3926a-c401-41d4-91cd-78e4f1c3c04c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114912025-04-30T06:43:21.686ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
114922025-04-30T06:43:21.686ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
114932025-04-30T06:43:21.689ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114942025-04-30T06:43:21.689ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114952025-04-30T06:43:21.689ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114962025-04-30T06:43:21.689ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
114972025-04-30T06:43:21.690ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114982025-04-30T06:43:21.691ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:59760 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
114992025-04-30T06:43:21.691ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
115002025-04-30T06:43:21.691ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
115012025-04-30T06:43:21.691ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
115022025-04-30T06:43:21.691ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
115032025-04-30T06:43:21.693ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
115042025-04-30T06:43:21.694ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
115052025-04-30T06:43:21.696ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 107506 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = f0a3926a-c401-41d4-91cd-78e4f1c3c04c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115062025-04-30T06:43:21.699ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = 127.0.0.1:47175 method = GET remote_addr = 127.0.0.1:55312 req_id = 8c22fdc8-fd67-4847-80e2-a06bcf4eeaea uri = /58f2365f-b720-4ec1-ad50-a68fadcacb1c
115072025-04-30T06:43:21.700ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:59760 req_id = 26ea8642-289b-47cb-a845-92a6352cce98 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115082025-04-30T06:43:21.700ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0a3926a-c401-41d4-91cd-78e4f1c3c04c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
115092025-04-30T06:43:21.700ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
115102025-04-30T06:43:21.706ZINFOtest_snapshot_expunge (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 6442 local_addr = 127.0.0.1:47175 method = GET remote_addr = 127.0.0.1:55312 req_id = 8c22fdc8-fd67-4847-80e2-a06bcf4eeaea response_code = 200 uri = /58f2365f-b720-4ec1-ad50-a68fadcacb1c
115112025-04-30T06:43:21.706ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 6382 local_addr = [::1]:51119 method = GET remote_addr = [::1]:59760 req_id = 26ea8642-289b-47cb-a845-92a6352cce98 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115122025-04-30T06:43:21.706ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115132025-04-30T06:43:21.722ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115142025-04-30T06:43:21.722ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115152025-04-30T06:43:21.723ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
115162025-04-30T06:43:21.729ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115172025-04-30T06:43:21.729ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
115182025-04-30T06:43:21.729ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
115192025-04-30T06:43:21.744ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 019234d0-dcce-4c87-8eaf-085ccfd77de0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
115202025-04-30T06:43:21.745ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
115212025-04-30T06:43:21.747ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 019234d0-dcce-4c87-8eaf-085ccfd77de0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
115222025-04-30T06:43:21.747ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 019234d0-dcce-4c87-8eaf-085ccfd77de0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
115232025-04-30T06:43:21.750ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 019234d0-dcce-4c87-8eaf-085ccfd77de0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
115242025-04-30T06:43:21.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "26ea8642-289b-47cb-a845-92a6352cce98", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
115252025-04-30T06:43:21.755ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115262025-04-30T06:43:21.755ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115272025-04-30T06:43:21.755ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
115282025-04-30T06:43:21.757ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115292025-04-30T06:43:21.757ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115302025-04-30T06:43:21.758ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115312025-04-30T06:43:21.758ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115322025-04-30T06:43:21.758ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115332025-04-30T06:43:21.769ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
115342025-04-30T06:43:21.769ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
115352025-04-30T06:43:21.769ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
115362025-04-30T06:43:21.771ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115372025-04-30T06:43:21.771ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115382025-04-30T06:43:21.774ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115392025-04-30T06:43:21.774ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
115402025-04-30T06:43:21.774ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115412025-04-30T06:43:21.775ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
115422025-04-30T06:43:21.775ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
115432025-04-30T06:43:21.778ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 33949 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 019234d0-dcce-4c87-8eaf-085ccfd77de0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
115442025-04-30T06:43:21.778ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "019234d0-dcce-4c87-8eaf-085ccfd77de0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
115452025-04-30T06:43:21.778ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
115462025-04-30T06:43:21.778ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115472025-04-30T06:43:21.778ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
115482025-04-30T06:43:21.778ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115492025-04-30T06:43:21.779ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115502025-04-30T06:43:21.779ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115512025-04-30T06:43:21.779ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115522025-04-30T06:43:21.782ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115532025-04-30T06:43:21.790ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115542025-04-30T06:43:21.791ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115552025-04-30T06:43:21.791ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115562025-04-30T06:43:21.792ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115572025-04-30T06:43:21.795ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
115582025-04-30T06:43:21.843ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
115592025-04-30T06:43:21.872ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
115602025-04-30T06:43:21.872ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
115612025-04-30T06:43:21.872ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 613.351162ms iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
115622025-04-30T06:43:21.872ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
115632025-04-30T06:43:21.872ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115642025-04-30T06:43:21.873ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115652025-04-30T06:43:21.873ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115662025-04-30T06:43:21.873ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115672025-04-30T06:43:21.874ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115682025-04-30T06:43:21.877ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115692025-04-30T06:43:21.883ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115702025-04-30T06:43:21.884ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115712025-04-30T06:43:21.884ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115722025-04-30T06:43:21.885ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115732025-04-30T06:43:21.908ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115742025-04-30T06:43:21.922ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115752025-04-30T06:43:21.922ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115762025-04-30T06:43:21.922ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
115772025-04-30T06:43:21.924ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115782025-04-30T06:43:21.925ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
115792025-04-30T06:43:21.925ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
115802025-04-30T06:43:21.926ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 03 da 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56190 req_id = efdfba21-9579-4340-9371-cc38691410fe
115812025-04-30T06:43:21.926ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 986,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56190 req_id = efdfba21-9579-4340-9371-cc38691410fe
115822025-04-30T06:43:21.926ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
115832025-04-30T06:43:21.926ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115842025-04-30T06:43:21.926ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56190 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = efdfba21-9579-4340-9371-cc38691410fe
115852025-04-30T06:43:21.927ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4266, tv_nsec: 726415110 } })
115862025-04-30T06:43:21.927ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
115872025-04-30T06:43:21.928ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 58 81 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e X...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62947 req_id = 84027187-ad56-4bde-8073-fbd2e5a7cef3
115882025-04-30T06:43:21.928ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22657,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62947 req_id = 84027187-ad56-4bde-8073-fbd2e5a7cef3
115892025-04-30T06:43:21.928ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115902025-04-30T06:43:21.928ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62947 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 84027187-ad56-4bde-8073-fbd2e5a7cef3
115912025-04-30T06:43:21.929ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115922025-04-30T06:43:21.939ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115932025-04-30T06:43:21.939ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115942025-04-30T06:43:21.939ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
115952025-04-30T06:43:21.939ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115962025-04-30T06:43:22.048ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
115972025-04-30T06:43:22.075ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a3 5e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .^..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50022 req_id = 07bcc8ab-5d17-408f-8a5a-b7aab7181e88
115982025-04-30T06:43:22.075ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41822,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50022 req_id = 07bcc8ab-5d17-408f-8a5a-b7aab7181e88
115992025-04-30T06:43:22.076ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
116002025-04-30T06:43:22.076ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41822, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50022 req_id = 07bcc8ab-5d17-408f-8a5a-b7aab7181e88
116012025-04-30T06:43:22.076ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116022025-04-30T06:43:22.076ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
116032025-04-30T06:43:22.077ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116042025-04-30T06:43:22.077ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
116052025-04-30T06:43:22.077ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
116062025-04-30T06:43:22.078ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
116072025-04-30T06:43:22.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "862633f2-545f-4ca6-93a4-ec662b7ef561", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116082025-04-30T06:43:22.118ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
116092025-04-30T06:43:22.137ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e78706d6-a21f-4ca5-9451-70543306178d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116102025-04-30T06:43:22.140ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
116112025-04-30T06:43:22.151ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
116122025-04-30T06:43:22.152ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 16 4a 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .J..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:51795 req_id = b2b524bf-c8e9-4a37-bdeb-0f23adc6787c
116132025-04-30T06:43:22.152ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5706,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51795 req_id = b2b524bf-c8e9-4a37-bdeb-0f23adc6787c
116142025-04-30T06:43:22.152ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
116152025-04-30T06:43:22.153ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116162025-04-30T06:43:22.153ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51795 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b2b524bf-c8e9-4a37-bdeb-0f23adc6787c
116172025-04-30T06:43:22.154ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4266, tv_nsec: 953388720 } })
116182025-04-30T06:43:22.154ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
116192025-04-30T06:43:22.155ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d4 67 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .g..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58590 req_id = 41c979d7-709f-46ab-bcd1-0b79a2bd4cca
116202025-04-30T06:43:22.155ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54375,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58590 req_id = 41c979d7-709f-46ab-bcd1-0b79a2bd4cca
116212025-04-30T06:43:22.155ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116222025-04-30T06:43:22.155ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58590 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 41c979d7-709f-46ab-bcd1-0b79a2bd4cca
116232025-04-30T06:43:22.155ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
116242025-04-30T06:43:22.232ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
116252025-04-30T06:43:22.232ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
116262025-04-30T06:43:22.233ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
116272025-04-30T06:43:22.233ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
116282025-04-30T06:43:22.235ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
116292025-04-30T06:43:22.236ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9015e423-94a7-47aa-aa99-fbbe0b94bbc0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116302025-04-30T06:43:22.236ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
116312025-04-30T06:43:22.240ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e390234e-ff99-430a-aa68-e396e2e71f53", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116322025-04-30T06:43:22.240ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
116332025-04-30T06:43:22.241ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
116342025-04-30T06:43:22.255ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
116352025-04-30T06:43:22.257ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7b ce 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 {............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55176 req_id = 68a34de8-2d7b-4855-9ad2-52fa3e0c1a38
116362025-04-30T06:43:22.257ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31694,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55176 req_id = 68a34de8-2d7b-4855-9ad2-52fa3e0c1a38
116372025-04-30T06:43:22.257ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
116382025-04-30T06:43:22.257ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116392025-04-30T06:43:22.257ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55176 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 68a34de8-2d7b-4855-9ad2-52fa3e0c1a38
116402025-04-30T06:43:22.258ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4267, tv_nsec: 57172723 } })
116412025-04-30T06:43:22.258ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
116422025-04-30T06:43:22.259ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 81 7a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .z..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65282 req_id = 627214f4-44f0-4106-8da9-9ad23609edbf
116432025-04-30T06:43:22.259ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33146,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65282 req_id = 627214f4-44f0-4106-8da9-9ad23609edbf
116442025-04-30T06:43:22.259ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116452025-04-30T06:43:22.259ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65282 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 627214f4-44f0-4106-8da9-9ad23609edbf
116462025-04-30T06:43:22.259ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
116472025-04-30T06:43:22.346ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
116482025-04-30T06:43:22.347ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
116492025-04-30T06:43:22.347ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
116502025-04-30T06:43:22.347ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
116512025-04-30T06:43:22.349ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
116522025-04-30T06:43:22.350ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116532025-04-30T06:43:22.350ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116542025-04-30T06:43:22.350ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116552025-04-30T06:43:22.350ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
116562025-04-30T06:43:22.351ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116572025-04-30T06:43:22.352ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22ee9e20-89cf-4281-9285-1ca7559b512f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116582025-04-30T06:43:22.352ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
116592025-04-30T06:43:22.354ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116602025-04-30T06:43:22.361ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116612025-04-30T06:43:22.361ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116622025-04-30T06:43:22.361ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
116632025-04-30T06:43:22.362ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116642025-04-30T06:43:22.368ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
116652025-04-30T06:43:22.369ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c3 46 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .F..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59041 req_id = 4b6ca2f0-4e62-4430-bb55-502a4b32a7e0
116662025-04-30T06:43:22.370ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49990,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59041 req_id = 4b6ca2f0-4e62-4430-bb55-502a4b32a7e0
116672025-04-30T06:43:22.370ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
116682025-04-30T06:43:22.371ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116692025-04-30T06:43:22.378ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59041 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4b6ca2f0-4e62-4430-bb55-502a4b32a7e0
116702025-04-30T06:43:22.378ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116712025-04-30T06:43:22.378ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
116722025-04-30T06:43:22.378ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116732025-04-30T06:43:22.379ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4267, tv_nsec: 177581880 } })
116742025-04-30T06:43:22.379ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
116752025-04-30T06:43:22.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "feac2b84-05d2-4f89-812c-0ce039de4af9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116762025-04-30T06:43:22.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
116772025-04-30T06:43:22.380ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: df b5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62673 req_id = 8c6e883a-0831-4be8-b7fb-44f15a02a3c6
116782025-04-30T06:43:22.380ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57269,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62673 req_id = 8c6e883a-0831-4be8-b7fb-44f15a02a3c6
116792025-04-30T06:43:22.380ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116802025-04-30T06:43:22.380ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62673 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8c6e883a-0831-4be8-b7fb-44f15a02a3c6
116812025-04-30T06:43:22.383ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
116822025-04-30T06:43:22.454ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
116832025-04-30T06:43:22.454ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
116842025-04-30T06:43:22.455ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
116852025-04-30T06:43:22.455ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
116862025-04-30T06:43:22.459ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
116872025-04-30T06:43:22.462ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e70a602-155e-4faa-b83f-48c5166a6ac1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116882025-04-30T06:43:22.463ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
116892025-04-30T06:43:22.476ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1dfa53c-f133-4932-9a11-39fd265296ef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116902025-04-30T06:43:22.476ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
116912025-04-30T06:43:22.479ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e17af3d-b67b-4675-8bc9-b2e05b3e4fc1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116922025-04-30T06:43:22.479ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
116932025-04-30T06:43:22.481ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72d5fc1d-6cbb-4c69-8d22-fb2654037eda", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116942025-04-30T06:43:22.482ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
116952025-04-30T06:43:22.484ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ad8e4ab-b58d-4516-8e8f-ffc3f6d604e0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116962025-04-30T06:43:22.484ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
116972025-04-30T06:43:22.485ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a04fc83-e71f-44a3-b533-25d96d138aa7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
116982025-04-30T06:43:22.485ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
116992025-04-30T06:43:22.491ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a46bd767-8a5a-45d6-b8ea-ed716ddfee04", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117002025-04-30T06:43:22.492ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
117012025-04-30T06:43:22.505ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
117022025-04-30T06:43:22.507ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42ababd9-23a9-43ae-87c0-521b5f27e76d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117032025-04-30T06:43:22.507ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
117042025-04-30T06:43:22.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f26b80b8-78aa-4374-958f-e652203d3c59", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117052025-04-30T06:43:22.510ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
117062025-04-30T06:43:22.513ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57457216-86d2-4b24-95d8-fb48d3c68226", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117072025-04-30T06:43:22.513ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
117082025-04-30T06:43:22.515ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a742d414-525e-4bdf-b9e8-f882e4ae98d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117092025-04-30T06:43:22.515ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
117102025-04-30T06:43:22.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f096be9-6207-43fa-93ce-4bb0e3c9b0eb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117112025-04-30T06:43:22.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
117122025-04-30T06:43:22.520ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a574b02-f1ac-4c7d-b856-652399b601d2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117132025-04-30T06:43:22.520ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
117142025-04-30T06:43:22.521ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a63e680-5578-4438-b491-dee35a0eb700", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117152025-04-30T06:43:22.521ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
117162025-04-30T06:43:22.522ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "074e7edd-cfdb-4356-8735-ddc4eb603d64", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117172025-04-30T06:43:22.522ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
117182025-04-30T06:43:22.523ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c90054e-5804-4812-80e1-07ea5591fffa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117192025-04-30T06:43:22.524ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
117202025-04-30T06:43:22.524ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "920900ea-d1b3-40f0-b171-9642737b8aaa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117212025-04-30T06:43:22.525ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
117222025-04-30T06:43:22.526ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdaca2ef-342d-4a03-9b70-5b034d3d98c1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117232025-04-30T06:43:22.526ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
117242025-04-30T06:43:22.528ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea5bf0ef-6bf7-47b7-8dae-fdbe3d9ddc1b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117252025-04-30T06:43:22.528ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
117262025-04-30T06:43:22.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8efeed99-4fbf-4862-9071-46dac36c9fd0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117272025-04-30T06:43:22.530ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
117282025-04-30T06:43:22.556ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117292025-04-30T06:43:22.556ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117302025-04-30T06:43:22.556ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117312025-04-30T06:43:22.556ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
117322025-04-30T06:43:22.569ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117332025-04-30T06:43:22.569ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d853fcf-533a-4a88-83ff-bcc198c1ab8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117342025-04-30T06:43:22.569ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
117352025-04-30T06:43:22.572ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117362025-04-30T06:43:22.581ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117372025-04-30T06:43:22.581ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117382025-04-30T06:43:22.581ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
117392025-04-30T06:43:22.582ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117402025-04-30T06:43:22.582ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f4408d-d174-46d1-ac3f-285d7daa054d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117412025-04-30T06:43:22.583ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
117422025-04-30T06:43:22.585ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117432025-04-30T06:43:22.585ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
117442025-04-30T06:43:22.587ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117452025-04-30T06:43:22.592ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e100123e-6833-4136-983a-464612417979", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117462025-04-30T06:43:22.592ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
117472025-04-30T06:43:22.599ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac729ec7-d22d-4971-8157-b253c24bd757", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117482025-04-30T06:43:22.599ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
117492025-04-30T06:43:22.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c2ba33a-54e6-46db-bce7-516b84044755", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117502025-04-30T06:43:22.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
117512025-04-30T06:43:22.603ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7337564e-0427-4287-9b27-49e91f35e3ff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117522025-04-30T06:43:22.603ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
117532025-04-30T06:43:22.604ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53da62a2-32b5-4dac-a18f-c8d4710ebb1f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117542025-04-30T06:43:22.605ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
117552025-04-30T06:43:22.607ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6412384d-729e-4efe-a51f-f1107f6ab7ad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117562025-04-30T06:43:22.607ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
117572025-04-30T06:43:22.611ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2efcc600-c3d8-436f-893b-f7446a1ec96a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117582025-04-30T06:43:22.612ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
117592025-04-30T06:43:22.617ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a590f045-29bf-4b70-8452-383d337a26b1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117602025-04-30T06:43:22.617ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
117612025-04-30T06:43:22.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a2b81c1-1dc4-42f6-bd9a-fc72350e65f2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117622025-04-30T06:43:22.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
117632025-04-30T06:43:22.649ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
117642025-04-30T06:43:22.651ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad4113cd-de28-418a-b853-778af95dea7d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117652025-04-30T06:43:22.651ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
117662025-04-30T06:43:22.654ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fe46cb2-1cbf-479e-9413-07e2d96f7063", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117672025-04-30T06:43:22.654ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
117682025-04-30T06:43:22.656ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0438d339-a004-41e7-af65-7282c6e9862d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117692025-04-30T06:43:22.656ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
117702025-04-30T06:43:22.658ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "893fa1fe-9597-4cf7-b67c-96857b341c85", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117712025-04-30T06:43:22.658ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
117722025-04-30T06:43:22.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f95aff0a-db7b-4aa3-8eb7-8f8e723997b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117732025-04-30T06:43:22.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
117742025-04-30T06:43:22.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "756e2ac4-d6f1-46eb-ac58-277e3dd8b26e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117752025-04-30T06:43:22.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
117762025-04-30T06:43:22.662ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c6dd88e-3a78-403e-822d-31d024cb80a6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117772025-04-30T06:43:22.662ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
117782025-04-30T06:43:22.663ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f65490a2-61b0-40d0-a1a6-03e9481673b3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
117792025-04-30T06:43:22.663ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117802025-04-30T06:43:22.664ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117812025-04-30T06:43:22.664ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117822025-04-30T06:43:22.664ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117832025-04-30T06:43:22.664ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117842025-04-30T06:43:22.680ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117852025-04-30T06:43:22.686ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117862025-04-30T06:43:22.686ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117872025-04-30T06:43:22.686ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117882025-04-30T06:43:22.686ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117892025-04-30T06:43:22.686ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117902025-04-30T06:43:22.687ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117912025-04-30T06:43:22.688ZDEBGtest_snapshot_expunge (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117922025-04-30T06:43:22.688ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117932025-04-30T06:43:22.688ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
117942025-04-30T06:43:22.711ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
117952025-04-30T06:43:22.732ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117962025-04-30T06:43:22.745ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117972025-04-30T06:43:22.745ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117982025-04-30T06:43:22.745ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117992025-04-30T06:43:22.746ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118002025-04-30T06:43:22.746ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118012025-04-30T06:43:22.746ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118022025-04-30T06:43:22.746ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118032025-04-30T06:43:22.747ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
118042025-04-30T06:43:22.747ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118052025-04-30T06:43:22.748ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118062025-04-30T06:43:22.748ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118072025-04-30T06:43:22.752ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 12 table_name = fields_string
118082025-04-30T06:43:22.752ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
118092025-04-30T06:43:22.752ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118102025-04-30T06:43:22.754ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118112025-04-30T06:43:22.754ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118122025-04-30T06:43:22.754ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118132025-04-30T06:43:22.828ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
118142025-04-30T06:43:22.828ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
118152025-04-30T06:43:22.829ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118162025-04-30T06:43:22.834ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118172025-04-30T06:43:22.835ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118182025-04-30T06:43:22.835ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
118192025-04-30T06:43:22.835ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118202025-04-30T06:43:22.835ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:59690 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
118212025-04-30T06:43:22.839ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
118222025-04-30T06:43:22.845ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 5 table_name = fields_u16
118232025-04-30T06:43:22.845ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
118242025-04-30T06:43:22.845ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:59690 req_id = 21fe827d-e7ae-4291-a8fa-a0c386610629 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
118252025-04-30T06:43:22.845ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 457 local_addr = [::1]:51119 method = GET remote_addr = [::1]:59690 req_id = 21fe827d-e7ae-4291-a8fa-a0c386610629 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
118262025-04-30T06:43:22.846ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118272025-04-30T06:43:22.846ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
118282025-04-30T06:43:22.847ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118292025-04-30T06:43:22.847ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "21fe827d-e7ae-4291-a8fa-a0c386610629", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
118302025-04-30T06:43:22.851ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
118312025-04-30T06:43:22.851ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
118322025-04-30T06:43:22.851ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
118332025-04-30T06:43:22.854ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
118342025-04-30T06:43:22.854ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118352025-04-30T06:43:22.854ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118362025-04-30T06:43:22.854ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118372025-04-30T06:43:22.854ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118382025-04-30T06:43:22.862ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 5 table_name = fields_uuid
118392025-04-30T06:43:22.862ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
118402025-04-30T06:43:22.864ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118412025-04-30T06:43:22.868ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
118422025-04-30T06:43:22.868ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118432025-04-30T06:43:22.868ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118442025-04-30T06:43:22.869ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118452025-04-30T06:43:22.870ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 5 table_name = measurements_histogramu64
118462025-04-30T06:43:22.870ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
118472025-04-30T06:43:22.876ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
118482025-04-30T06:43:22.876ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
118492025-04-30T06:43:22.946ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
118502025-04-30T06:43:22.958ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
118512025-04-30T06:43:22.958ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
118522025-04-30T06:43:22.958ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.086069573s iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
118532025-04-30T06:43:22.958ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
118542025-04-30T06:43:22.958ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
118552025-04-30T06:43:22.959ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118562025-04-30T06:43:22.959ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118572025-04-30T06:43:22.959ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118582025-04-30T06:43:22.961ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118592025-04-30T06:43:22.967ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118602025-04-30T06:43:22.971ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
118612025-04-30T06:43:22.971ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118622025-04-30T06:43:22.971ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118632025-04-30T06:43:22.972ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118642025-04-30T06:43:23.004ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118652025-04-30T06:43:23.004ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118662025-04-30T06:43:23.004ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118672025-04-30T06:43:23.004ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
118682025-04-30T06:43:23.005ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118692025-04-30T06:43:23.006ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
118702025-04-30T06:43:23.006ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
118712025-04-30T06:43:23.009ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5b 83 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 [............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58862 req_id = fc2908c2-5932-4464-beda-e87e34863b1f
118722025-04-30T06:43:23.009ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23427,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58862 req_id = fc2908c2-5932-4464-beda-e87e34863b1f
118732025-04-30T06:43:23.009ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
118742025-04-30T06:43:23.009ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118752025-04-30T06:43:23.010ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58862 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = fc2908c2-5932-4464-beda-e87e34863b1f
118762025-04-30T06:43:23.010ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118772025-04-30T06:43:23.017ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118782025-04-30T06:43:23.017ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118792025-04-30T06:43:23.017ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
118802025-04-30T06:43:23.018ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118812025-04-30T06:43:23.019ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4267, tv_nsec: 818424948 } })
118822025-04-30T06:43:23.019ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
118832025-04-30T06:43:23.025ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e8 0e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51859 req_id = 350f6e13-679c-4920-a725-6d5e997959e8
118842025-04-30T06:43:23.026ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59406,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51859 req_id = 350f6e13-679c-4920-a725-6d5e997959e8
118852025-04-30T06:43:23.026ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118862025-04-30T06:43:23.026ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51859 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 350f6e13-679c-4920-a725-6d5e997959e8
118872025-04-30T06:43:23.026ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118882025-04-30T06:43:23.026ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
118892025-04-30T06:43:23.027ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118902025-04-30T06:43:23.099ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
118912025-04-30T06:43:23.120ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 09 25 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .%..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51348 req_id = 0c383de2-4185-4fa2-8dbb-0cfc7d366412
118922025-04-30T06:43:23.120ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2341,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51348 req_id = 0c383de2-4185-4fa2-8dbb-0cfc7d366412
118932025-04-30T06:43:23.121ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
118942025-04-30T06:43:23.121ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2341, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51348 req_id = 0c383de2-4185-4fa2-8dbb-0cfc7d366412
118952025-04-30T06:43:23.123ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
118962025-04-30T06:43:23.123ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
118972025-04-30T06:43:23.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
118982025-04-30T06:43:23.136ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5697e8e-8e5c-445f-b3a1-b01bd813ac5b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
118992025-04-30T06:43:23.137ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
119002025-04-30T06:43:23.149ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52f74154-756c-4f4b-8140-7ed14d5dda77", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119012025-04-30T06:43:23.150ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
119022025-04-30T06:43:23.166ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1de4160-9095-40a1-8fcc-4923a7424fcf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119032025-04-30T06:43:23.166ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
119042025-04-30T06:43:23.171ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53ed5bb2-87e3-4d91-9945-c890754584de", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119052025-04-30T06:43:23.171ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
119062025-04-30T06:43:23.173ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60d45099-150e-41ca-8ca4-fa71a1c9b19e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119072025-04-30T06:43:23.173ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
119082025-04-30T06:43:23.175ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1677666e-435b-4b35-9518-6a6db7386604", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119092025-04-30T06:43:23.175ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
119102025-04-30T06:43:23.177ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed1360fd-32e9-4893-8bcf-77af70c5deb6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119112025-04-30T06:43:23.177ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
119122025-04-30T06:43:23.181ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "700e716e-26b1-4266-8b65-13da149e4c0d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119132025-04-30T06:43:23.181ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
119142025-04-30T06:43:23.181ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f38b2511-4bf2-4a1d-95ca-8a16981149f9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119152025-04-30T06:43:23.181ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
119162025-04-30T06:43:23.181ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "453e99c5-f2e3-43c9-98a7-2974d0e87b4a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119172025-04-30T06:43:23.181ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
119182025-04-30T06:43:23.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce102ec1-7906-43c3-9bd9-b27db1e36d9a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119192025-04-30T06:43:23.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
119202025-04-30T06:43:23.190ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da02e148-b69a-4fd4-bbdc-4f79b3cb462d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119212025-04-30T06:43:23.190ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
119222025-04-30T06:43:23.191ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
119232025-04-30T06:43:23.193ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dd554e3-9067-4d44-b9f2-b6e23ad202cd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119242025-04-30T06:43:23.194ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
119252025-04-30T06:43:23.198ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cf368fb-f63e-4912-a385-cb45c8ab4142", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119262025-04-30T06:43:23.198ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
119272025-04-30T06:43:23.200ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ca784c1-a892-41c3-8c2b-d2986ac229d6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119282025-04-30T06:43:23.200ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
119292025-04-30T06:43:23.203ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac61a533-cf8c-48a0-a31b-0a31f1fc9826", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119302025-04-30T06:43:23.203ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
119312025-04-30T06:43:23.206ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb0765d-1c5e-4803-b475-1c1da294d6d2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119322025-04-30T06:43:23.207ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
119332025-04-30T06:43:23.224ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d939baea-5197-40b2-aa4f-25b0013ab551", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119342025-04-30T06:43:23.224ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
119352025-04-30T06:43:23.246ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119362025-04-30T06:43:23.246ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119372025-04-30T06:43:23.246ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119382025-04-30T06:43:23.247ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
119392025-04-30T06:43:23.247ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119402025-04-30T06:43:23.256ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119412025-04-30T06:43:23.256ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119422025-04-30T06:43:23.256ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119432025-04-30T06:43:23.256ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
119442025-04-30T06:43:23.256ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119452025-04-30T06:43:23.256ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6da3d5b9-63cb-4cdc-ba9f-c2b951792477", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119462025-04-30T06:43:23.256ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
119472025-04-30T06:43:23.264ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119482025-04-30T06:43:23.264ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
119492025-04-30T06:43:23.265ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119502025-04-30T06:43:23.268ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6acd956-6aaf-4493-ade8-5bd953a7a653", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119512025-04-30T06:43:23.268ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
119522025-04-30T06:43:23.269ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a488d94f-aa2c-481e-af5e-ee63378c92e9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119532025-04-30T06:43:23.269ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
119542025-04-30T06:43:23.270ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d6a94ec-9e3b-4257-affd-786cf2f761b3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119552025-04-30T06:43:23.270ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
119562025-04-30T06:43:23.275ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64c1262a-2798-4dba-a5be-072331599d12", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119572025-04-30T06:43:23.275ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
119582025-04-30T06:43:23.277ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f906565-6b40-4463-b3fe-2daf7a953751", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119592025-04-30T06:43:23.285ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
119602025-04-30T06:43:23.324ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d89c8918-883b-4d12-908e-7aba1f7df766", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119612025-04-30T06:43:23.324ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
119622025-04-30T06:43:23.358ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90dbd87d-6cfe-49ba-8a3a-4fd0df1ab88e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119632025-04-30T06:43:23.371ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
119642025-04-30T06:43:23.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5256896a-4c88-4cad-95f1-3f9b7c4b87ce", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119652025-04-30T06:43:23.384ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
119662025-04-30T06:43:23.398ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
119672025-04-30T06:43:23.424ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed6bdccd-55e4-455f-a0d4-35bb08f57616", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119682025-04-30T06:43:23.424ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
119692025-04-30T06:43:23.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "258a1250-a513-42bb-8444-f4367da14eaf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119702025-04-30T06:43:23.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
119712025-04-30T06:43:23.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fad8bc1-ce7b-4eac-9e46-79bb781d09b7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119722025-04-30T06:43:23.435ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
119732025-04-30T06:43:23.437ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1abda1e4-3da0-4f33-b027-4dbaef5f92fe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119742025-04-30T06:43:23.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
119752025-04-30T06:43:23.445ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "591d9008-8d61-4a17-ad98-8e1e13c57470", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119762025-04-30T06:43:23.445ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
119772025-04-30T06:43:23.449ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "173b144f-633d-4717-aacb-ff88db9c4a1a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119782025-04-30T06:43:23.450ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
119792025-04-30T06:43:23.450ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119802025-04-30T06:43:23.450ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119812025-04-30T06:43:23.450ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119822025-04-30T06:43:23.450ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
119832025-04-30T06:43:23.452ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119842025-04-30T06:43:23.457ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119852025-04-30T06:43:23.465ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119862025-04-30T06:43:23.465ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119872025-04-30T06:43:23.465ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
119882025-04-30T06:43:23.466ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119892025-04-30T06:43:23.466ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcb845c5-0b12-4e8a-b035-d41ed87fd873", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119902025-04-30T06:43:23.466ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
119912025-04-30T06:43:23.469ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119922025-04-30T06:43:23.469ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
119932025-04-30T06:43:23.470ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119942025-04-30T06:43:23.474ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59d49dbc-e8ad-4c24-a34a-59fc87810faa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119952025-04-30T06:43:23.475ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
119962025-04-30T06:43:23.479ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92e3ee91-83d3-4b66-9d45-a9dadeb78488", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119972025-04-30T06:43:23.479ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
119982025-04-30T06:43:23.489ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94280998-16f6-4fd0-bb4a-85de8132152d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
119992025-04-30T06:43:23.489ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
120002025-04-30T06:43:23.493ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cca2098-7286-468c-b78d-ac9fd943b8c0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120012025-04-30T06:43:23.493ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
120022025-04-30T06:43:23.496ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad5f872c-ed13-4634-a748-c1c8d695c20b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120032025-04-30T06:43:23.497ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
120042025-04-30T06:43:23.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1006f4d2-2302-4d71-943f-335fadce309e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120052025-04-30T06:43:23.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
120062025-04-30T06:43:23.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23ed42bc-54bc-4ed7-8854-fae6e31b36f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120072025-04-30T06:43:23.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
120082025-04-30T06:43:23.511ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
120092025-04-30T06:43:23.511ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77aaa123-f600-4a24-bd68-99726680014f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120102025-04-30T06:43:23.511ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
120112025-04-30T06:43:23.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b3469f2-50d5-444c-911d-e7ef9e575fb0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120122025-04-30T06:43:23.518ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
120132025-04-30T06:43:23.533ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95fa945d-078d-4a86-a999-f80e9df5ad11", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120142025-04-30T06:43:23.533ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
120152025-04-30T06:43:23.534ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "380fb681-b9e2-49f4-8828-1ce9964127a2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120162025-04-30T06:43:23.535ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
120172025-04-30T06:43:23.535ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120182025-04-30T06:43:23.535ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120192025-04-30T06:43:23.535ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120202025-04-30T06:43:23.536ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120212025-04-30T06:43:23.546ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120222025-04-30T06:43:23.556ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
120232025-04-30T06:43:23.556ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
120242025-04-30T06:43:23.556ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120252025-04-30T06:43:23.556ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120262025-04-30T06:43:23.556ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120272025-04-30T06:43:23.557ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120282025-04-30T06:43:23.560ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120292025-04-30T06:43:23.565ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
120302025-04-30T06:43:23.565ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120312025-04-30T06:43:23.565ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120322025-04-30T06:43:23.575ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120332025-04-30T06:43:23.578ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120342025-04-30T06:43:23.578ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120352025-04-30T06:43:23.578ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120362025-04-30T06:43:23.578ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
120372025-04-30T06:43:23.579ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120382025-04-30T06:43:23.584ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120392025-04-30T06:43:23.600ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120402025-04-30T06:43:23.600ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120412025-04-30T06:43:23.600ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
120422025-04-30T06:43:23.600ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120432025-04-30T06:43:23.600ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
120442025-04-30T06:43:23.601ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120452025-04-30T06:43:23.679ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
120462025-04-30T06:43:23.679ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
120472025-04-30T06:43:23.681ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:50326 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
120482025-04-30T06:43:23.682ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4ae3d1ee-e8f9-4ea5-8a20-8d936794fb69 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120492025-04-30T06:43:23.682ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4ae3d1ee-e8f9-4ea5-8a20-8d936794fb69 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120502025-04-30T06:43:23.682ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4ae3d1ee-e8f9-4ea5-8a20-8d936794fb69 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120512025-04-30T06:43:23.687ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4ae3d1ee-e8f9-4ea5-8a20-8d936794fb69 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120522025-04-30T06:43:23.688ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120532025-04-30T06:43:23.689ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
120542025-04-30T06:43:23.691ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120552025-04-30T06:43:23.691ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:50326 req_id = 5dd1678c-d5cb-449e-be04-d2869a4f2ee9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120562025-04-30T06:43:23.691ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120572025-04-30T06:43:23.691ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120582025-04-30T06:43:23.694ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 2712 local_addr = [::1]:51119 method = GET remote_addr = [::1]:50326 req_id = 5dd1678c-d5cb-449e-be04-d2869a4f2ee9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120592025-04-30T06:43:23.694ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
120602025-04-30T06:43:23.695ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
120612025-04-30T06:43:23.696ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 14186 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 4ae3d1ee-e8f9-4ea5-8a20-8d936794fb69 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120622025-04-30T06:43:23.698ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ae3d1ee-e8f9-4ea5-8a20-8d936794fb69", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120632025-04-30T06:43:23.699ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120642025-04-30T06:43:23.699ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dd1678c-d5cb-449e-be04-d2869a4f2ee9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120652025-04-30T06:43:23.702ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120662025-04-30T06:43:23.702ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120672025-04-30T06:43:23.702ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
120682025-04-30T06:43:23.705ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120692025-04-30T06:43:23.706ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120702025-04-30T06:43:23.706ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120712025-04-30T06:43:23.706ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120722025-04-30T06:43:23.706ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120732025-04-30T06:43:23.708ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = ff62aaf5-ad5f-4557-b509-db61c3d2f145 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120742025-04-30T06:43:23.708ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = ff62aaf5-ad5f-4557-b509-db61c3d2f145 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120752025-04-30T06:43:23.708ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = ff62aaf5-ad5f-4557-b509-db61c3d2f145 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120762025-04-30T06:43:23.710ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = ff62aaf5-ad5f-4557-b509-db61c3d2f145 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120772025-04-30T06:43:23.713ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
120782025-04-30T06:43:23.713ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
120792025-04-30T06:43:23.716ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120802025-04-30T06:43:23.721ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120812025-04-30T06:43:23.722ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120822025-04-30T06:43:23.722ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120832025-04-30T06:43:23.723ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120842025-04-30T06:43:23.725ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 17032 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = ff62aaf5-ad5f-4557-b509-db61c3d2f145 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120852025-04-30T06:43:23.728ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120862025-04-30T06:43:23.728ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff62aaf5-ad5f-4557-b509-db61c3d2f145", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
120872025-04-30T06:43:23.729ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
120882025-04-30T06:43:23.729ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120892025-04-30T06:43:23.732ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
120902025-04-30T06:43:23.732ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120912025-04-30T06:43:23.736ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
120922025-04-30T06:43:23.736ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120932025-04-30T06:43:23.742ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120942025-04-30T06:43:23.743ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120952025-04-30T06:43:23.743ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120962025-04-30T06:43:23.743ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120972025-04-30T06:43:23.768ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
120982025-04-30T06:43:23.768ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
120992025-04-30T06:43:23.768ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 809.38151ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
121002025-04-30T06:43:23.768ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
121012025-04-30T06:43:23.768ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
121022025-04-30T06:43:23.768ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121032025-04-30T06:43:23.768ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121042025-04-30T06:43:23.768ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121052025-04-30T06:43:23.769ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121062025-04-30T06:43:23.771ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
121072025-04-30T06:43:23.779ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121082025-04-30T06:43:23.786ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
121092025-04-30T06:43:23.786ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121102025-04-30T06:43:23.786ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121112025-04-30T06:43:23.786ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121122025-04-30T06:43:23.791ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
121132025-04-30T06:43:23.791ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
121142025-04-30T06:43:23.792ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e4 e4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51248 req_id = 9f119a77-c0ae-4fd6-a17e-532d068d093b
121152025-04-30T06:43:23.792ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58596,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51248 req_id = 9f119a77-c0ae-4fd6-a17e-532d068d093b
121162025-04-30T06:43:23.793ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
121172025-04-30T06:43:23.793ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121182025-04-30T06:43:23.793ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51248 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9f119a77-c0ae-4fd6-a17e-532d068d093b
121192025-04-30T06:43:23.793ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4268, tv_nsec: 592793868 } })
121202025-04-30T06:43:23.793ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
121212025-04-30T06:43:23.794ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
121222025-04-30T06:43:23.794ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a2 ec 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60380 req_id = a89ed74e-66ea-4993-ba4c-2cb33161340b
121232025-04-30T06:43:23.794ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41708,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60380 req_id = a89ed74e-66ea-4993-ba4c-2cb33161340b
121242025-04-30T06:43:23.795ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121252025-04-30T06:43:23.795ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60380 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a89ed74e-66ea-4993-ba4c-2cb33161340b
121262025-04-30T06:43:23.864ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
121272025-04-30T06:43:23.865ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121282025-04-30T06:43:23.865ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121292025-04-30T06:43:23.865ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121302025-04-30T06:43:23.865ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
121312025-04-30T06:43:23.867ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121322025-04-30T06:43:23.867ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f3 4a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .J..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65465 req_id = 3e4b36ae-e95a-4e0b-bb9a-fd85c20ddb59
121332025-04-30T06:43:23.867ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62282,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65465 req_id = 3e4b36ae-e95a-4e0b-bb9a-fd85c20ddb59
121342025-04-30T06:43:23.867ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
121352025-04-30T06:43:23.867ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62282, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65465 req_id = 3e4b36ae-e95a-4e0b-bb9a-fd85c20ddb59
121362025-04-30T06:43:23.868ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
121372025-04-30T06:43:23.868ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
121382025-04-30T06:43:23.868ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
121392025-04-30T06:43:23.870ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121402025-04-30T06:43:23.880ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121412025-04-30T06:43:23.880ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121422025-04-30T06:43:23.880ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
121432025-04-30T06:43:23.881ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121442025-04-30T06:43:23.882ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "98cc20ab-f297-434f-b7e5-678a5c142d08", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121452025-04-30T06:43:23.883ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
121462025-04-30T06:43:23.883ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121472025-04-30T06:43:23.883ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
121482025-04-30T06:43:23.886ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121492025-04-30T06:43:23.890ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43205857-5cba-45e7-bb7a-5b017ed78ed3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121502025-04-30T06:43:23.891ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
121512025-04-30T06:43:23.897ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3be9c6df-25a2-4495-989b-e8879ac8accb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121522025-04-30T06:43:23.897ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
121532025-04-30T06:43:23.907ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8499062-5445-4993-a8e6-98531a029280", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121542025-04-30T06:43:23.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
121552025-04-30T06:43:23.917ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e92a5682-67fa-4a72-b97b-bace6fb04cc5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121562025-04-30T06:43:23.920ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
121572025-04-30T06:43:23.927ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35a6370e-a92e-40f7-b4f2-89251e1dcc2d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121582025-04-30T06:43:23.928ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
121592025-04-30T06:43:23.929ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
121602025-04-30T06:43:23.933ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4aab5be4-0448-4b0a-9c1e-e7ab5ace9187", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121612025-04-30T06:43:23.934ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
121622025-04-30T06:43:23.941ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1872bd58-a7ed-4e41-aacd-01d2c32a8925", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121632025-04-30T06:43:23.942ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
121642025-04-30T06:43:23.943ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "922d5900-4dc8-4313-a0b6-93b25910d3bf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121652025-04-30T06:43:23.944ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
121662025-04-30T06:43:23.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "428063f1-fd6b-423b-88d2-32049a2d4192", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121672025-04-30T06:43:23.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
121682025-04-30T06:43:23.947ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d785d2f8-988e-4398-ac1a-ff42f27236dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121692025-04-30T06:43:23.947ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
121702025-04-30T06:43:23.948ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0920628b-b083-4a0f-b50a-3fc7790b4844", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121712025-04-30T06:43:23.949ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
121722025-04-30T06:43:23.951ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e59767b2-5473-40c4-852e-b47a4f6c38e7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121732025-04-30T06:43:23.951ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
121742025-04-30T06:43:23.953ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ce3bc2e-4193-42ff-85a6-0c9a30e03f8e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121752025-04-30T06:43:23.953ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
121762025-04-30T06:43:23.959ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eae3c3e-6e85-4248-ac10-84685075579b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121772025-04-30T06:43:23.959ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
121782025-04-30T06:43:23.961ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfb7a426-9a47-4efb-9ce1-9917a5528dea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121792025-04-30T06:43:23.961ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
121802025-04-30T06:43:23.963ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a33a726-fab2-455c-8198-e7d59b09b7c8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121812025-04-30T06:43:23.963ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
121822025-04-30T06:43:23.965ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e249dbb1-a4c0-4dad-a83a-c81170354c44", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121832025-04-30T06:43:23.965ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
121842025-04-30T06:43:23.968ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e4bc297-2f57-48c9-bc3f-e623155720c6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121852025-04-30T06:43:23.968ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
121862025-04-30T06:43:23.969ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78e8635e-c904-40e1-9241-a0d60d091b3f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121872025-04-30T06:43:23.969ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
121882025-04-30T06:43:23.971ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d6790b1-d59c-4ece-8714-edc799b7564c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121892025-04-30T06:43:23.971ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
121902025-04-30T06:43:23.975ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8476d4ce-f1c6-4fea-8eef-99e536c7c25a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121912025-04-30T06:43:23.975ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
121922025-04-30T06:43:23.976ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aad274f-c6b9-4413-8136-6e452f87b4f4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121932025-04-30T06:43:23.976ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
121942025-04-30T06:43:23.978ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dba92c3-8624-4c7a-b30d-09b4bd2ccfec", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
121952025-04-30T06:43:23.978ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
121962025-04-30T06:43:23.981ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121972025-04-30T06:43:23.981ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121982025-04-30T06:43:23.981ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121992025-04-30T06:43:23.981ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122002025-04-30T06:43:23.981ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122012025-04-30T06:43:23.986ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122022025-04-30T06:43:23.990ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122032025-04-30T06:43:23.990ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122042025-04-30T06:43:23.991ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122052025-04-30T06:43:23.991ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122062025-04-30T06:43:23.996ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122072025-04-30T06:43:23.996ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122082025-04-30T06:43:23.997ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122092025-04-30T06:43:24.009ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aef9718-a67c-48e3-9dd7-dd5dacf8208d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122102025-04-30T06:43:24.009ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
122112025-04-30T06:43:24.050ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39cfccc5-561f-47d3-9ab5-09d561cf45d1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122122025-04-30T06:43:24.050ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
122132025-04-30T06:43:24.053ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e620aee2-8f9c-44bf-8518-0b238caa4e1d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122142025-04-30T06:43:24.053ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
122152025-04-30T06:43:24.058ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c2aa77f-8de1-4813-b848-d721d60a4d13", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122162025-04-30T06:43:24.058ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
122172025-04-30T06:43:24.059ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
122182025-04-30T06:43:24.062ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7c6c1ed-8a24-4a1f-ac1d-94979f02f5ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122192025-04-30T06:43:24.063ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
122202025-04-30T06:43:24.064ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "741a27ac-2f92-4901-ba51-a3fcfb1dd458", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122212025-04-30T06:43:24.065ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
122222025-04-30T06:43:24.066ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa8d6749-9a33-4a4c-b671-0f596413534e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122232025-04-30T06:43:24.066ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
122242025-04-30T06:43:24.068ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56a4de6e-1514-4c82-860d-40d81d2d108a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122252025-04-30T06:43:24.068ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
122262025-04-30T06:43:24.069ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cd0e0ce-6915-411e-80e5-cd4808028564", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122272025-04-30T06:43:24.069ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
122282025-04-30T06:43:24.070ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2808b226-a40b-45de-a578-c62789809d63", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122292025-04-30T06:43:24.070ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
122302025-04-30T06:43:24.071ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ca82395-ca33-4dbd-bf69-3f0360c3f582", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122312025-04-30T06:43:24.072ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
122322025-04-30T06:43:24.075ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ef90dff-a261-485f-8ed4-618de357cc6e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122332025-04-30T06:43:24.076ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
122342025-04-30T06:43:24.079ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "148dd342-cb39-419a-a0b4-6297df928c4b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122352025-04-30T06:43:24.079ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
122362025-04-30T06:43:24.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b013f31-a833-4a90-abc7-2d88500a91f6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122372025-04-30T06:43:24.082ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
122382025-04-30T06:43:24.085ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "241bf02e-4b61-435e-b3b6-03215d73baa4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122392025-04-30T06:43:24.085ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
122402025-04-30T06:43:24.087ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e82a766f-746c-41d3-a1c0-99e2949f321b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122412025-04-30T06:43:24.088ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
122422025-04-30T06:43:24.090ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c18161e-5c58-4569-a5f1-d84eb33049a6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122432025-04-30T06:43:24.091ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
122442025-04-30T06:43:24.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "485f5487-5e68-4d06-8fe0-736567d773ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122452025-04-30T06:43:24.093ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
122462025-04-30T06:43:24.104ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f29c4c64-ceb2-403b-98e2-b26b3185dc2b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122472025-04-30T06:43:24.104ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
122482025-04-30T06:43:24.106ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec8647cc-ed24-4de8-a2bf-3f69e6b404a1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122492025-04-30T06:43:24.107ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
122502025-04-30T06:43:24.111ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b29223f9-7bd1-4acf-b60c-ad6acaad682c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122512025-04-30T06:43:24.111ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122522025-04-30T06:43:24.111ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122532025-04-30T06:43:24.111ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122542025-04-30T06:43:24.111ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122552025-04-30T06:43:24.111ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122562025-04-30T06:43:24.111ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122572025-04-30T06:43:24.111ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122582025-04-30T06:43:24.111ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122592025-04-30T06:43:24.111ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122602025-04-30T06:43:24.112ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122612025-04-30T06:43:24.116ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122622025-04-30T06:43:24.123ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122632025-04-30T06:43:24.123ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122642025-04-30T06:43:24.123ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122652025-04-30T06:43:24.124ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122662025-04-30T06:43:24.125ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122672025-04-30T06:43:24.134ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122682025-04-30T06:43:24.135ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122692025-04-30T06:43:24.135ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122702025-04-30T06:43:24.135ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122712025-04-30T06:43:24.136ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122722025-04-30T06:43:24.137ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122732025-04-30T06:43:24.140ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122742025-04-30T06:43:24.140ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122752025-04-30T06:43:24.141ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122762025-04-30T06:43:24.145ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122772025-04-30T06:43:24.162ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122782025-04-30T06:43:24.162ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122792025-04-30T06:43:24.162ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122802025-04-30T06:43:24.162ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122812025-04-30T06:43:24.237ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
122822025-04-30T06:43:24.237ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
122832025-04-30T06:43:24.237ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:46746 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
122842025-04-30T06:43:24.239ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
122852025-04-30T06:43:24.241ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:46746 req_id = cbde21db-3aaa-47f3-b234-fa979a0487e8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122862025-04-30T06:43:24.242ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 678 local_addr = [::1]:51119 method = GET remote_addr = [::1]:46746 req_id = cbde21db-3aaa-47f3-b234-fa979a0487e8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122872025-04-30T06:43:24.242ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c7 20 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 . ..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:60073 req_id = f10bdfde-8ecb-46d5-90a0-1d1ae0bf7464
122882025-04-30T06:43:24.242ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50976,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60073 req_id = f10bdfde-8ecb-46d5-90a0-1d1ae0bf7464
122892025-04-30T06:43:24.243ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
122902025-04-30T06:43:24.243ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122912025-04-30T06:43:24.243ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60073 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f10bdfde-8ecb-46d5-90a0-1d1ae0bf7464
122922025-04-30T06:43:24.243ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbde21db-3aaa-47f3-b234-fa979a0487e8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
122932025-04-30T06:43:24.246ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122942025-04-30T06:43:24.246ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122952025-04-30T06:43:24.246ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
122962025-04-30T06:43:24.252ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122972025-04-30T06:43:24.252ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122982025-04-30T06:43:24.252ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122992025-04-30T06:43:24.252ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123002025-04-30T06:43:24.253ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123012025-04-30T06:43:24.255ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4269, tv_nsec: 54575407 } })
123022025-04-30T06:43:24.255ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
123032025-04-30T06:43:24.257ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d0 4a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .J..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54081 req_id = 1297a422-4307-4cb5-917b-00203814b419
123042025-04-30T06:43:24.257ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53322,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54081 req_id = 1297a422-4307-4cb5-917b-00203814b419
123052025-04-30T06:43:24.257ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123062025-04-30T06:43:24.258ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54081 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1297a422-4307-4cb5-917b-00203814b419
123072025-04-30T06:43:24.258ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123082025-04-30T06:43:24.263ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
123092025-04-30T06:43:24.263ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123102025-04-30T06:43:24.263ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123112025-04-30T06:43:24.264ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123122025-04-30T06:43:24.265ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
123132025-04-30T06:43:24.344ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
123142025-04-30T06:43:24.346ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
123152025-04-30T06:43:24.346ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
123162025-04-30T06:43:24.346ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
123172025-04-30T06:43:24.351ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
123182025-04-30T06:43:24.352ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
123192025-04-30T06:43:24.354ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 2f 87 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 /............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:58691 req_id = ab1892ce-0306-4a6d-b494-c660ea7a475d
123202025-04-30T06:43:24.354ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12167,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58691 req_id = ab1892ce-0306-4a6d-b494-c660ea7a475d
123212025-04-30T06:43:24.354ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
123222025-04-30T06:43:24.355ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123232025-04-30T06:43:24.355ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58691 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ab1892ce-0306-4a6d-b494-c660ea7a475d
123242025-04-30T06:43:24.356ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4269, tv_nsec: 154991620 } })
123252025-04-30T06:43:24.356ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
123262025-04-30T06:43:24.356ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: cb 44 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .D..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55747 req_id = f8189557-94ec-487c-9e4b-84282ae6035d
123272025-04-30T06:43:24.356ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52036,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55747 req_id = f8189557-94ec-487c-9e4b-84282ae6035d
123282025-04-30T06:43:24.357ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123292025-04-30T06:43:24.357ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55747 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f8189557-94ec-487c-9e4b-84282ae6035d
123302025-04-30T06:43:24.357ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
123312025-04-30T06:43:24.431ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
123322025-04-30T06:43:24.432ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
123332025-04-30T06:43:24.438ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
123342025-04-30T06:43:24.438ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
123352025-04-30T06:43:24.439ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
123362025-04-30T06:43:24.461ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
123372025-04-30T06:43:24.462ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ee e3 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55208 req_id = 8b9ae083-303c-492b-9379-e3ffafab8361
123382025-04-30T06:43:24.463ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61155,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55208 req_id = 8b9ae083-303c-492b-9379-e3ffafab8361
123392025-04-30T06:43:24.463ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
123402025-04-30T06:43:24.463ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123412025-04-30T06:43:24.463ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55208 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8b9ae083-303c-492b-9379-e3ffafab8361
123422025-04-30T06:43:24.464ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4269, tv_nsec: 263129929 } })
123432025-04-30T06:43:24.465ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
123442025-04-30T06:43:24.468ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 54 49 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e TI..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63450 req_id = 4ca21a31-2894-4f73-ba3b-7b8c11ef30b5
123452025-04-30T06:43:24.468ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21577,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63450 req_id = 4ca21a31-2894-4f73-ba3b-7b8c11ef30b5
123462025-04-30T06:43:24.470ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123472025-04-30T06:43:24.470ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63450 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4ca21a31-2894-4f73-ba3b-7b8c11ef30b5
123482025-04-30T06:43:24.471ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
123492025-04-30T06:43:24.576ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
123502025-04-30T06:43:24.576ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
123512025-04-30T06:43:24.576ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
123522025-04-30T06:43:24.576ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
123532025-04-30T06:43:24.578ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
123542025-04-30T06:43:24.605ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
123552025-04-30T06:43:24.609ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
123562025-04-30T06:43:24.609ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
123572025-04-30T06:43:24.609ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 841.168533ms iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
123582025-04-30T06:43:24.610ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
123592025-04-30T06:43:24.610ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
123602025-04-30T06:43:24.610ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123612025-04-30T06:43:24.610ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123622025-04-30T06:43:24.610ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123632025-04-30T06:43:24.613ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123642025-04-30T06:43:24.624ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123652025-04-30T06:43:24.632ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
123662025-04-30T06:43:24.632ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123672025-04-30T06:43:24.632ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123682025-04-30T06:43:24.633ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123692025-04-30T06:43:24.639ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
123702025-04-30T06:43:24.639ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
123712025-04-30T06:43:24.642ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 95 d3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62701 req_id = ab58eec4-9302-4ece-87f2-2be138617a6e
123722025-04-30T06:43:24.642ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38355,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62701 req_id = ab58eec4-9302-4ece-87f2-2be138617a6e
123732025-04-30T06:43:24.642ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
123742025-04-30T06:43:24.642ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123752025-04-30T06:43:24.642ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62701 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ab58eec4-9302-4ece-87f2-2be138617a6e
123762025-04-30T06:43:24.643ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4269, tv_nsec: 442294993 } })
123772025-04-30T06:43:24.643ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
123782025-04-30T06:43:24.643ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 47 83 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e G...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54363 req_id = 21810f93-8d42-4d61-943a-d53631354eb2
123792025-04-30T06:43:24.643ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18307,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54363 req_id = 21810f93-8d42-4d61-943a-d53631354eb2
123802025-04-30T06:43:24.643ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123812025-04-30T06:43:24.644ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54363 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 21810f93-8d42-4d61-943a-d53631354eb2
123822025-04-30T06:43:24.710ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
123832025-04-30T06:43:24.711ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123842025-04-30T06:43:24.711ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123852025-04-30T06:43:24.711ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123862025-04-30T06:43:24.711ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
123872025-04-30T06:43:24.712ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123882025-04-30T06:43:24.712ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
123892025-04-30T06:43:24.712ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
123902025-04-30T06:43:24.712ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
123912025-04-30T06:43:24.713ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
123922025-04-30T06:43:24.716ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 35 1f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 5............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54152 req_id = 743ba709-8f65-4367-a99f-5b4658713395
123932025-04-30T06:43:24.716ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13599,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54152 req_id = 743ba709-8f65-4367-a99f-5b4658713395
123942025-04-30T06:43:24.716ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
123952025-04-30T06:43:24.716ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13599, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54152 req_id = 743ba709-8f65-4367-a99f-5b4658713395
123962025-04-30T06:43:24.717ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123972025-04-30T06:43:24.725ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123982025-04-30T06:43:24.725ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123992025-04-30T06:43:24.725ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
124002025-04-30T06:43:24.726ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124012025-04-30T06:43:24.726ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
124022025-04-30T06:43:24.726ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
124032025-04-30T06:43:24.726ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
124042025-04-30T06:43:24.727ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
124052025-04-30T06:43:24.727ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
124062025-04-30T06:43:24.728ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124072025-04-30T06:43:24.728ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124082025-04-30T06:43:24.732ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124092025-04-30T06:43:24.732ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
124102025-04-30T06:43:24.733ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124112025-04-30T06:43:24.733ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
124122025-04-30T06:43:24.733ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124132025-04-30T06:43:24.734ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124142025-04-30T06:43:24.735ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124152025-04-30T06:43:24.736ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124162025-04-30T06:43:24.736ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124172025-04-30T06:43:24.737ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1965980-ef87-44c0-bbc8-d59744193ad1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124182025-04-30T06:43:24.737ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
124192025-04-30T06:43:24.742ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9ce2eaf-117b-407b-a37e-e4a47e1cfa88", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124202025-04-30T06:43:24.743ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
124212025-04-30T06:43:24.747ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "121020ac-820f-4f3a-94f4-806cc8a34f45", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124222025-04-30T06:43:24.747ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
124232025-04-30T06:43:24.753ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "960331c9-3dfb-456a-ad18-d2194420afa0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124242025-04-30T06:43:24.753ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
124252025-04-30T06:43:24.759ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e19bc8de-40b6-4a8c-ac5b-4088360037f6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124262025-04-30T06:43:24.759ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
124272025-04-30T06:43:24.762ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff470c11-179c-4c0b-926b-e7b449264396", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124282025-04-30T06:43:24.763ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
124292025-04-30T06:43:24.765ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c7b5f8c-2992-478b-91be-f8ecde62431b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124302025-04-30T06:43:24.765ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
124312025-04-30T06:43:24.773ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
124322025-04-30T06:43:24.773ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c8cf726-54db-4415-9da5-7f1939a43650", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124332025-04-30T06:43:24.774ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
124342025-04-30T06:43:24.776ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09d1621e-58cb-4ef8-ba40-7e156c03da20", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124352025-04-30T06:43:24.776ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
124362025-04-30T06:43:24.777ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa5befe4-807b-4199-b7db-5ad3b12adbbd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124372025-04-30T06:43:24.777ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
124382025-04-30T06:43:24.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35003089-6234-4329-b69b-0ab109449ab9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124392025-04-30T06:43:24.779ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
124402025-04-30T06:43:24.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b3a11b8-ab20-466d-a814-a46309649bce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124412025-04-30T06:43:24.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
124422025-04-30T06:43:24.781ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63a299c5-6c88-4a5b-9937-1acac96f5079", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124432025-04-30T06:43:24.782ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
124442025-04-30T06:43:24.789ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e41af6-1dda-4b40-bdff-1e069cd17224", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124452025-04-30T06:43:24.789ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
124462025-04-30T06:43:24.798ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a358b794-4044-4813-9a6c-a44017ff4509", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124472025-04-30T06:43:24.798ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
124482025-04-30T06:43:24.800ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a81d4a0-053d-4b49-9ea2-13771852b26d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124492025-04-30T06:43:24.800ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
124502025-04-30T06:43:24.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d32433c-4ae2-4ed3-9226-be912d98a152", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124512025-04-30T06:43:24.803ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
124522025-04-30T06:43:24.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5307c13f-6ce2-4d26-9d68-65c0e49e137f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124532025-04-30T06:43:24.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
124542025-04-30T06:43:24.808ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89bb2ed2-613d-46cd-a2f1-13c4abadc08f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124552025-04-30T06:43:24.808ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
124562025-04-30T06:43:24.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6716de13-8dc5-4800-aefd-310d946ef766", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124572025-04-30T06:43:24.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
124582025-04-30T06:43:24.811ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce45e27-e9f7-411f-9645-8ded024e30af", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124592025-04-30T06:43:24.812ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
124602025-04-30T06:43:24.813ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
124612025-04-30T06:43:24.813ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e59e29e-9201-4c66-b3a4-292145b45a60", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124622025-04-30T06:43:24.814ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
124632025-04-30T06:43:24.816ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4e5328c-e513-4ca0-9518-a01e2505ffb9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124642025-04-30T06:43:24.816ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
124652025-04-30T06:43:24.819ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83342b19-4f16-48e4-abb5-dfbdb7382e80", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124662025-04-30T06:43:24.819ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
124672025-04-30T06:43:24.825ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124682025-04-30T06:43:24.825ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124692025-04-30T06:43:24.826ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124702025-04-30T06:43:24.826ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
124712025-04-30T06:43:24.827ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124722025-04-30T06:43:24.828ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2d8bd9f-c408-425d-8f5f-82cd28f05f83", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124732025-04-30T06:43:24.828ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
124742025-04-30T06:43:24.831ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124752025-04-30T06:43:24.838ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124762025-04-30T06:43:24.839ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124772025-04-30T06:43:24.839ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
124782025-04-30T06:43:24.841ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124792025-04-30T06:43:24.844ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124802025-04-30T06:43:24.844ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
124812025-04-30T06:43:24.845ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124822025-04-30T06:43:24.846ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b924377-6db4-4c17-9473-21c536fe9a58", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124832025-04-30T06:43:24.846ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
124842025-04-30T06:43:24.851ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22a4fde2-e6ef-4516-ae13-040d0389db7e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124852025-04-30T06:43:24.851ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
124862025-04-30T06:43:24.864ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "707d22ee-0cd2-42c2-aeb7-4d3b1955ba93", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124872025-04-30T06:43:24.864ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
124882025-04-30T06:43:24.870ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47d96195-d031-4655-8147-f7af34113d22", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124892025-04-30T06:43:24.870ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
124902025-04-30T06:43:24.872ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74e36804-254a-4b22-b68a-37b45ff1f734", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124912025-04-30T06:43:24.872ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
124922025-04-30T06:43:24.874ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af2b617-e81e-4442-8063-8981c3fc2e99", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124932025-04-30T06:43:24.874ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
124942025-04-30T06:43:24.876ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11afbbc9-6182-4797-8394-65cda98d4285", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
124952025-04-30T06:43:24.876ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
124962025-04-30T06:43:24.886ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c5bddbc-c37a-48b6-996e-8f2ed221b119", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
124972025-04-30T06:43:24.886ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
124982025-04-30T06:43:24.887ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
124992025-04-30T06:43:24.888ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f882bb8d-ba97-47ab-994a-b7ed197999f3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125002025-04-30T06:43:24.888ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
125012025-04-30T06:43:24.897ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba1de224-454e-4efa-8be7-630f1555a324", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125022025-04-30T06:43:24.897ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
125032025-04-30T06:43:24.899ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffacac4e-5a6b-47a3-8253-7b6b9ebfd242", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125042025-04-30T06:43:24.899ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
125052025-04-30T06:43:24.904ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e63738e-7db7-4479-8a9c-c44a5bfddc49", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125062025-04-30T06:43:24.904ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
125072025-04-30T06:43:24.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69802ab7-1b13-4912-a4d2-c764ba8ac3a4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125082025-04-30T06:43:24.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
125092025-04-30T06:43:24.910ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcd6eba4-45e5-44aa-be8c-9c9094410f34", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125102025-04-30T06:43:24.910ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
125112025-04-30T06:43:24.912ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fbf2e36-aa19-404c-b576-a9842f3ba9fe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125122025-04-30T06:43:24.912ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
125132025-04-30T06:43:24.914ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "528427d9-715d-4cc8-a283-62bafe5ba375", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125142025-04-30T06:43:24.914ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
125152025-04-30T06:43:24.915ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d4ecb4b-90fe-42ff-88fd-558491813f71", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125162025-04-30T06:43:24.916ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
125172025-04-30T06:43:24.917ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8e84894-7a00-469f-abc2-8adcce610a30", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125182025-04-30T06:43:24.917ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
125192025-04-30T06:43:24.918ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f4adf71-6343-4ace-9e11-5da16f00fda9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125202025-04-30T06:43:24.918ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
125212025-04-30T06:43:24.923ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4e3b260-c883-4f0b-a1c6-e327b18aefb7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125222025-04-30T06:43:24.930ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
125232025-04-30T06:43:24.930ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125242025-04-30T06:43:24.930ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125252025-04-30T06:43:24.930ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125262025-04-30T06:43:24.940ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125272025-04-30T06:43:24.941ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125282025-04-30T06:43:24.941ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125292025-04-30T06:43:24.941ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125302025-04-30T06:43:24.941ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
125312025-04-30T06:43:24.942ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125322025-04-30T06:43:24.943ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125332025-04-30T06:43:24.961ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
125342025-04-30T06:43:24.961ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
125352025-04-30T06:43:24.961ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125362025-04-30T06:43:24.961ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125372025-04-30T06:43:24.961ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125382025-04-30T06:43:24.962ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125392025-04-30T06:43:24.967ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125402025-04-30T06:43:25.005ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
125412025-04-30T06:43:25.005ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125422025-04-30T06:43:25.005ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125432025-04-30T06:43:25.006ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125442025-04-30T06:43:25.006ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125452025-04-30T06:43:25.047ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125462025-04-30T06:43:25.047ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125472025-04-30T06:43:25.047ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
125482025-04-30T06:43:25.047ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125492025-04-30T06:43:25.152ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
125502025-04-30T06:43:25.152ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
125512025-04-30T06:43:25.153ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125522025-04-30T06:43:25.153ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
125532025-04-30T06:43:25.154ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125542025-04-30T06:43:25.154ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:63396 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
125552025-04-30T06:43:25.156ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:63396 req_id = 570cb74b-a1b7-4d02-bb1e-f25a3de0b5d7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
125562025-04-30T06:43:25.160ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 3897 local_addr = [::1]:51119 method = GET remote_addr = [::1]:63396 req_id = 570cb74b-a1b7-4d02-bb1e-f25a3de0b5d7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
125572025-04-30T06:43:25.161ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "570cb74b-a1b7-4d02-bb1e-f25a3de0b5d7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
125582025-04-30T06:43:25.164ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
125592025-04-30T06:43:25.164ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
125602025-04-30T06:43:25.164ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
125612025-04-30T06:43:25.166ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
125622025-04-30T06:43:25.166ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125632025-04-30T06:43:25.166ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125642025-04-30T06:43:25.166ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125652025-04-30T06:43:25.167ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125662025-04-30T06:43:25.177ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125672025-04-30T06:43:25.182ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
125682025-04-30T06:43:25.183ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125692025-04-30T06:43:25.184ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125702025-04-30T06:43:25.186ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125712025-04-30T06:43:25.248ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
125722025-04-30T06:43:25.261ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
125732025-04-30T06:43:25.261ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
125742025-04-30T06:43:25.261ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 651.292588ms iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
125752025-04-30T06:43:25.261ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
125762025-04-30T06:43:25.261ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
125772025-04-30T06:43:25.261ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125782025-04-30T06:43:25.261ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125792025-04-30T06:43:25.261ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125802025-04-30T06:43:25.263ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125812025-04-30T06:43:25.266ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125822025-04-30T06:43:25.270ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
125832025-04-30T06:43:25.270ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125842025-04-30T06:43:25.270ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125852025-04-30T06:43:25.271ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125862025-04-30T06:43:25.277ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
125872025-04-30T06:43:25.277ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
125882025-04-30T06:43:25.278ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bb 70 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .p..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55014 req_id = 5eb2b6e2-f861-43d7-8bf8-4782169dd6a2
125892025-04-30T06:43:25.278ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47984,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55014 req_id = 5eb2b6e2-f861-43d7-8bf8-4782169dd6a2
125902025-04-30T06:43:25.279ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
125912025-04-30T06:43:25.279ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125922025-04-30T06:43:25.279ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55014 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5eb2b6e2-f861-43d7-8bf8-4782169dd6a2
125932025-04-30T06:43:25.279ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4270, tv_nsec: 78738756 } })
125942025-04-30T06:43:25.279ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
125952025-04-30T06:43:25.280ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 52 35 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e R5..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64729 req_id = 48c9687a-e884-448d-831a-8562b0241906
125962025-04-30T06:43:25.280ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21045,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64729 req_id = 48c9687a-e884-448d-831a-8562b0241906
125972025-04-30T06:43:25.280ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125982025-04-30T06:43:25.280ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64729 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 48c9687a-e884-448d-831a-8562b0241906
125992025-04-30T06:43:25.336ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
126002025-04-30T06:43:25.337ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126012025-04-30T06:43:25.337ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126022025-04-30T06:43:25.337ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126032025-04-30T06:43:25.337ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
126042025-04-30T06:43:25.338ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126052025-04-30T06:43:25.339ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ba b3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57222 req_id = d1997156-a059-4143-b6e4-59458386614d
126062025-04-30T06:43:25.339ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47795,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57222 req_id = d1997156-a059-4143-b6e4-59458386614d
126072025-04-30T06:43:25.339ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
126082025-04-30T06:43:25.339ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47795, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57222 req_id = d1997156-a059-4143-b6e4-59458386614d
126092025-04-30T06:43:25.341ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
126102025-04-30T06:43:25.341ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
126112025-04-30T06:43:25.341ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
126122025-04-30T06:43:25.341ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126132025-04-30T06:43:25.357ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126142025-04-30T06:43:25.357ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126152025-04-30T06:43:25.357ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
126162025-04-30T06:43:25.357ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126172025-04-30T06:43:25.361ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f0fa725-4433-4d15-aa30-6a1c913c18de", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126182025-04-30T06:43:25.361ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
126192025-04-30T06:43:25.363ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126202025-04-30T06:43:25.363ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
126212025-04-30T06:43:25.363ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126222025-04-30T06:43:25.365ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1278d8a1-b488-44a9-bc8e-509f3f4bdf21", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126232025-04-30T06:43:25.365ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
126242025-04-30T06:43:25.368ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f41ba9d7-968e-4ff3-9d18-6a8c9c81ec3b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126252025-04-30T06:43:25.368ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
126262025-04-30T06:43:25.370ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc440ad2-f79c-443c-a000-b2bf2dd84da5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126272025-04-30T06:43:25.371ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
126282025-04-30T06:43:25.373ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "004d1f47-a022-4bd1-aa4b-3ce898f6183e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126292025-04-30T06:43:25.373ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
126302025-04-30T06:43:25.376ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "137aa4be-3913-4714-8fe2-86c537265947", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126312025-04-30T06:43:25.376ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
126322025-04-30T06:43:25.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f86842be-6040-4e65-8480-9ddc36160f79", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126332025-04-30T06:43:25.379ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
126342025-04-30T06:43:25.382ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b5f66b1-fc1a-42d9-8bbe-014148c6a1e8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126352025-04-30T06:43:25.383ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
126362025-04-30T06:43:25.385ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e62b6f1-ecf2-42ab-8f39-6f894f7d94b7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126372025-04-30T06:43:25.385ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
126382025-04-30T06:43:25.386ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "515ccf4c-793d-4724-ae25-21b51f84a488", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126392025-04-30T06:43:25.386ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
126402025-04-30T06:43:25.387ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3f3895a-f52d-45a4-98cd-c2c587f3af59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126412025-04-30T06:43:25.388ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
126422025-04-30T06:43:25.389ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2501b1b8-3ebe-4db4-88bb-2816eabf1e1b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126432025-04-30T06:43:25.389ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
126442025-04-30T06:43:25.394ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6742833-b8ab-4407-9ab8-5b85fc1fb311", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126452025-04-30T06:43:25.394ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
126462025-04-30T06:43:25.394ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
126472025-04-30T06:43:25.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e32d2e44-f86c-4f25-814e-e64f92c219b5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126482025-04-30T06:43:25.396ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
126492025-04-30T06:43:25.398ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "600d4cfe-9264-43f9-9844-4ff61efc9bbd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126502025-04-30T06:43:25.398ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
126512025-04-30T06:43:25.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5177c64-904c-4f2a-8484-ae5a41ddbd7b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126522025-04-30T06:43:25.401ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
126532025-04-30T06:43:25.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef744420-2ac5-4c68-83a5-b82556340f25", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126542025-04-30T06:43:25.404ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
126552025-04-30T06:43:25.409ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3468319e-c846-42e1-98a8-e60a511e650e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126562025-04-30T06:43:25.409ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
126572025-04-30T06:43:25.412ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a195447-b9aa-4b71-844f-91b6ed4feeaa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126582025-04-30T06:43:25.413ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
126592025-04-30T06:43:25.414ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "801cb67c-3a0a-430f-acea-5f476f2363ea", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126602025-04-30T06:43:25.415ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
126612025-04-30T06:43:25.420ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62bd815e-a7d7-4023-88a4-8e86cec884d3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126622025-04-30T06:43:25.420ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
126632025-04-30T06:43:25.421ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33701fa8-c486-4c2e-aa63-9c827df3de02", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126642025-04-30T06:43:25.423ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
126652025-04-30T06:43:25.424ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d573c5f6-9279-4f6c-8976-b2b6839e2077", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126662025-04-30T06:43:25.424ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
126672025-04-30T06:43:25.426ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1cb153e-612b-4930-aaba-44226587ee39", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126682025-04-30T06:43:25.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
126692025-04-30T06:43:25.432ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f63eb9cb-3e8e-40eb-9844-68f206125795", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126702025-04-30T06:43:25.433ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
126712025-04-30T06:43:25.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02c6328b-157e-4e38-9c39-c5d7e23d36ea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126722025-04-30T06:43:25.438ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
126732025-04-30T06:43:25.450ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126742025-04-30T06:43:25.450ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126752025-04-30T06:43:25.451ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126762025-04-30T06:43:25.451ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
126772025-04-30T06:43:25.458ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126782025-04-30T06:43:25.458ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "855b8f6f-5f4c-47c7-88de-87d6294c1d28", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126792025-04-30T06:43:25.459ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
126802025-04-30T06:43:25.461ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126812025-04-30T06:43:25.475ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126822025-04-30T06:43:25.475ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126832025-04-30T06:43:25.475ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
126842025-04-30T06:43:25.475ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126852025-04-30T06:43:25.478ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126862025-04-30T06:43:25.478ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
126872025-04-30T06:43:25.480ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126882025-04-30T06:43:25.483ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19431b47-1248-432e-89ba-5c8ad84b9ca5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126892025-04-30T06:43:25.483ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
126902025-04-30T06:43:25.489ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38532247-9e99-4628-9930-c4bfe7e6c3cf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126912025-04-30T06:43:25.489ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
126922025-04-30T06:43:25.494ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61addf80-8e8f-4c38-ba19-bb41482fcb13", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126932025-04-30T06:43:25.494ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
126942025-04-30T06:43:25.497ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47c3f0ad-6574-4a09-bca3-a5eca6994490", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126952025-04-30T06:43:25.497ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
126962025-04-30T06:43:25.500ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c8b68f1-84f2-488c-9557-951cbaca9726", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126972025-04-30T06:43:25.501ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
126982025-04-30T06:43:25.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3cac353-0586-40cd-8932-4a2ee1cdeece", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
126992025-04-30T06:43:25.503ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
127002025-04-30T06:43:25.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58888ba4-0015-4169-ac74-ae223de01b2f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127012025-04-30T06:43:25.506ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
127022025-04-30T06:43:25.509ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b22e7d7-f1ca-4863-ac22-9a99f3073845", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127032025-04-30T06:43:25.509ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
127042025-04-30T06:43:25.514ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
127052025-04-30T06:43:25.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3cfaf8d-b8b5-437a-918c-2487fbc56322", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127062025-04-30T06:43:25.517ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
127072025-04-30T06:43:25.520ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b17a7456-94ef-4cf2-b3e3-003cdabe818c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127082025-04-30T06:43:25.521ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
127092025-04-30T06:43:25.524ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4073f18-18a4-47fc-96f5-194cb765a377", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127102025-04-30T06:43:25.524ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
127112025-04-30T06:43:25.527ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8784eaa7-8677-4c26-8b9c-706ac6ae8433", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127122025-04-30T06:43:25.527ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
127132025-04-30T06:43:25.531ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1114419-cad9-485d-bc09-88ee04a6050d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127142025-04-30T06:43:25.531ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
127152025-04-30T06:43:25.535ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20d6f243-d58d-40c3-891f-1ca928b1031a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127162025-04-30T06:43:25.535ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
127172025-04-30T06:43:25.538ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd8bcce7-3d46-46f3-8e07-72d8da1fef79", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127182025-04-30T06:43:25.538ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
127192025-04-30T06:43:25.556ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fe42831-3256-4bae-a6db-4ad6564aa11f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127202025-04-30T06:43:25.556ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
127212025-04-30T06:43:25.558ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4524f3c9-4e72-4bed-a497-26e2b5c57d62", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127222025-04-30T06:43:25.559ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
127232025-04-30T06:43:25.561ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7df65dbf-5388-4744-a92a-ad03ae6ba4a2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127242025-04-30T06:43:25.561ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
127252025-04-30T06:43:25.561ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127262025-04-30T06:43:25.561ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127272025-04-30T06:43:25.561ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127282025-04-30T06:43:25.563ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127292025-04-30T06:43:25.566ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127302025-04-30T06:43:25.566ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127312025-04-30T06:43:25.567ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127322025-04-30T06:43:25.567ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
127332025-04-30T06:43:25.568ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127342025-04-30T06:43:25.589ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
127352025-04-30T06:43:25.589ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
127362025-04-30T06:43:25.600ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127372025-04-30T06:43:25.605ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
127382025-04-30T06:43:25.605ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
127392025-04-30T06:43:25.605ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127402025-04-30T06:43:25.606ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127412025-04-30T06:43:25.606ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127422025-04-30T06:43:25.606ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127432025-04-30T06:43:25.608ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 565a9e9b-4649-4deb-852f-85cc2c58c382 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
127442025-04-30T06:43:25.609ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 565a9e9b-4649-4deb-852f-85cc2c58c382 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
127452025-04-30T06:43:25.609ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 565a9e9b-4649-4deb-852f-85cc2c58c382 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
127462025-04-30T06:43:25.610ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 565a9e9b-4649-4deb-852f-85cc2c58c382 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
127472025-04-30T06:43:25.610ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127482025-04-30T06:43:25.640ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127492025-04-30T06:43:25.647ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127502025-04-30T06:43:25.647ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
127512025-04-30T06:43:25.648ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127522025-04-30T06:43:25.650ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127532025-04-30T06:43:25.656ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
127542025-04-30T06:43:25.656ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127552025-04-30T06:43:25.657ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127562025-04-30T06:43:25.657ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127572025-04-30T06:43:25.659ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 51045 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 565a9e9b-4649-4deb-852f-85cc2c58c382 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
127582025-04-30T06:43:25.674ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "565a9e9b-4649-4deb-852f-85cc2c58c382", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127592025-04-30T06:43:25.675ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
127602025-04-30T06:43:25.675ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127612025-04-30T06:43:25.675ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
127622025-04-30T06:43:25.675ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127632025-04-30T06:43:25.676ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = c01ba166-3ff2-4da3-be75-b99ef4793c80 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
127642025-04-30T06:43:25.676ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = c01ba166-3ff2-4da3-be75-b99ef4793c80 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
127652025-04-30T06:43:25.676ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = c01ba166-3ff2-4da3-be75-b99ef4793c80 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
127662025-04-30T06:43:25.676ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = c01ba166-3ff2-4da3-be75-b99ef4793c80 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
127672025-04-30T06:43:25.742ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
127682025-04-30T06:43:25.743ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
127692025-04-30T06:43:25.744ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:47535 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
127702025-04-30T06:43:25.744ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
127712025-04-30T06:43:25.744ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
127722025-04-30T06:43:25.744ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127732025-04-30T06:43:25.744ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127742025-04-30T06:43:25.745ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
127752025-04-30T06:43:25.746ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
127762025-04-30T06:43:25.748ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:47535 req_id = 189d5fb4-0151-4f99-b795-877d40c26d9c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
127772025-04-30T06:43:25.749ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 1561 local_addr = [::1]:51119 method = GET remote_addr = [::1]:47535 req_id = 189d5fb4-0151-4f99-b795-877d40c26d9c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
127782025-04-30T06:43:25.750ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 74173 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = c01ba166-3ff2-4da3-be75-b99ef4793c80 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
127792025-04-30T06:43:25.751ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "c01ba166-3ff2-4da3-be75-b99ef4793c80", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127802025-04-30T06:43:25.751ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
127812025-04-30T06:43:25.751ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127822025-04-30T06:43:25.751ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
127832025-04-30T06:43:25.751ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127842025-04-30T06:43:25.751ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127852025-04-30T06:43:25.752ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127862025-04-30T06:43:25.752ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127872025-04-30T06:43:25.752ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "189d5fb4-0151-4f99-b795-877d40c26d9c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
127882025-04-30T06:43:25.753ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
127892025-04-30T06:43:25.754ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
127902025-04-30T06:43:25.754ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
127912025-04-30T06:43:25.755ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
127922025-04-30T06:43:25.755ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127932025-04-30T06:43:25.755ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127942025-04-30T06:43:25.755ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127952025-04-30T06:43:25.756ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127962025-04-30T06:43:25.758ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
127972025-04-30T06:43:25.758ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
127982025-04-30T06:43:25.761ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127992025-04-30T06:43:25.768ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128002025-04-30T06:43:25.768ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128012025-04-30T06:43:25.768ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128022025-04-30T06:43:25.770ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128032025-04-30T06:43:25.771ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
128042025-04-30T06:43:25.771ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128052025-04-30T06:43:25.823ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
128062025-04-30T06:43:25.858ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
128072025-04-30T06:43:25.873ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
128082025-04-30T06:43:25.873ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
128092025-04-30T06:43:25.873ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 612.085631ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
128102025-04-30T06:43:25.874ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
128112025-04-30T06:43:25.874ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128122025-04-30T06:43:25.874ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128132025-04-30T06:43:25.874ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128142025-04-30T06:43:25.874ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128152025-04-30T06:43:25.874ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128162025-04-30T06:43:25.877ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128172025-04-30T06:43:25.885ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128182025-04-30T06:43:25.885ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128192025-04-30T06:43:25.885ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128202025-04-30T06:43:25.886ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128212025-04-30T06:43:25.887ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128222025-04-30T06:43:25.887ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128232025-04-30T06:43:25.887ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128242025-04-30T06:43:25.887ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
128252025-04-30T06:43:25.887ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128262025-04-30T06:43:25.900ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128272025-04-30T06:43:25.905ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128282025-04-30T06:43:25.905ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128292025-04-30T06:43:25.905ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
128302025-04-30T06:43:25.907ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128312025-04-30T06:43:25.908ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
128322025-04-30T06:43:25.908ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
128332025-04-30T06:43:25.909ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b9 a0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55367 req_id = f7f367d7-6afd-4789-b0af-28bdf182e708
128342025-04-30T06:43:25.909ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47520,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55367 req_id = f7f367d7-6afd-4789-b0af-28bdf182e708
128352025-04-30T06:43:25.909ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
128362025-04-30T06:43:25.910ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
128372025-04-30T06:43:25.910ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55367 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f7f367d7-6afd-4789-b0af-28bdf182e708
128382025-04-30T06:43:25.910ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4270, tv_nsec: 709475277 } })
128392025-04-30T06:43:25.910ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
128402025-04-30T06:43:25.910ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128412025-04-30T06:43:25.910ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
128422025-04-30T06:43:25.911ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128432025-04-30T06:43:25.912ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e8 6b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .k..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57209 req_id = 8e175d3d-b0ce-4139-8775-1621b94f5194
128442025-04-30T06:43:25.912ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59499,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57209 req_id = 8e175d3d-b0ce-4139-8775-1621b94f5194
128452025-04-30T06:43:25.912ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
128462025-04-30T06:43:25.912ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57209 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8e175d3d-b0ce-4139-8775-1621b94f5194
128472025-04-30T06:43:25.972ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
128482025-04-30T06:43:25.973ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f3 3f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .?..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55254 req_id = f78b8ca1-10d0-426d-8f0c-8d5e9d7cda34
128492025-04-30T06:43:25.973ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62271,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55254 req_id = f78b8ca1-10d0-426d-8f0c-8d5e9d7cda34
128502025-04-30T06:43:25.973ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
128512025-04-30T06:43:25.973ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62271, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55254 req_id = f78b8ca1-10d0-426d-8f0c-8d5e9d7cda34
128522025-04-30T06:43:25.974ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
128532025-04-30T06:43:25.974ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
128542025-04-30T06:43:25.974ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
128552025-04-30T06:43:25.976ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "954a35e6-80f5-4299-906c-9b48334a3ad1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128562025-04-30T06:43:25.976ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
128572025-04-30T06:43:25.978ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a74f96a8-71c4-4899-8813-5273160baee6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128582025-04-30T06:43:25.979ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
128592025-04-30T06:43:26.007ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f2028f6-4f42-4213-b75a-55ebbcc248e6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128602025-04-30T06:43:26.007ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
128612025-04-30T06:43:26.021ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "570d31f8-0c58-4fab-ac19-8357dd4f4fcd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128622025-04-30T06:43:26.022ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
128632025-04-30T06:43:26.024ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14e5a585-e0bd-4200-a609-b111e581bc3a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128642025-04-30T06:43:26.024ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
128652025-04-30T06:43:26.029ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
128662025-04-30T06:43:26.032ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58ea44a9-e8eb-4684-afa4-e2fa12790f62", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128672025-04-30T06:43:26.032ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
128682025-04-30T06:43:26.034ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee0d08d0-fde4-4775-9737-6d20e26c53a2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128692025-04-30T06:43:26.034ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
128702025-04-30T06:43:26.036ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98aa1f93-f3f3-41af-9fc7-e2b326f12ec7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128712025-04-30T06:43:26.036ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
128722025-04-30T06:43:26.037ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75db057e-dd33-4636-acb3-4d9cca8f47d4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128732025-04-30T06:43:26.037ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
128742025-04-30T06:43:26.038ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "974ee40f-698c-4668-83ec-c7985f0753c5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128752025-04-30T06:43:26.038ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
128762025-04-30T06:43:26.039ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7b29a6f-1ed1-4518-910e-96f0a59c1659", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128772025-04-30T06:43:26.040ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
128782025-04-30T06:43:26.041ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f312703-5707-425c-a133-0c5a44e8f62c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128792025-04-30T06:43:26.041ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
128802025-04-30T06:43:26.042ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8b55a45-0f7d-4cf0-8bd7-a6f4f546fedf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128812025-04-30T06:43:26.042ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
128822025-04-30T06:43:26.046ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce3af997-ac35-4273-b3bc-8068241acaa5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128832025-04-30T06:43:26.046ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
128842025-04-30T06:43:26.049ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d6e9fed-2ff8-48c2-8ee7-96bbad7b31a4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128852025-04-30T06:43:26.049ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
128862025-04-30T06:43:26.051ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39ace0b7-6f8d-40ba-9de4-c2ffb9b936a4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128872025-04-30T06:43:26.051ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
128882025-04-30T06:43:26.053ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f844aa-388c-45a4-834e-324859639758", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128892025-04-30T06:43:26.053ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
128902025-04-30T06:43:26.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d498620-d561-42f2-a9b0-5aa0c1410c29", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128912025-04-30T06:43:26.055ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
128922025-04-30T06:43:26.079ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bdd0312-cdab-4365-8074-515bb6736cd4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
128932025-04-30T06:43:26.080ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
128942025-04-30T06:43:26.088ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128952025-04-30T06:43:26.088ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128962025-04-30T06:43:26.088ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128972025-04-30T06:43:26.089ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
128982025-04-30T06:43:26.089ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128992025-04-30T06:43:26.091ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcc38c4a-e679-4329-818d-efa25ff9d697", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129002025-04-30T06:43:26.092ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
129012025-04-30T06:43:26.095ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129022025-04-30T06:43:26.107ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129032025-04-30T06:43:26.107ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129042025-04-30T06:43:26.108ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
129052025-04-30T06:43:26.108ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129062025-04-30T06:43:26.108ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a7eecb-651b-410a-9d15-f8f3ddcb9ee5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129072025-04-30T06:43:26.109ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
129082025-04-30T06:43:26.112ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "040995cc-65ce-4991-854f-01ff74ea8371", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129092025-04-30T06:43:26.114ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
129102025-04-30T06:43:26.114ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129112025-04-30T06:43:26.114ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
129122025-04-30T06:43:26.115ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129132025-04-30T06:43:26.116ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b38ac3d7-d3f5-4d8a-bf8f-cba536217f36", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129142025-04-30T06:43:26.116ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
129152025-04-30T06:43:26.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea8dbd20-d972-43fd-9507-7b8fa075d1db", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129162025-04-30T06:43:26.120ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
129172025-04-30T06:43:26.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61d98cbd-3101-4da8-b350-625c0f0864f3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129182025-04-30T06:43:26.123ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
129192025-04-30T06:43:26.127ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f49e0f71-78d3-45cc-b8f6-85c8c3b3679c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129202025-04-30T06:43:26.127ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
129212025-04-30T06:43:26.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2609e6e9-08f8-4f9c-977a-dac67cdb5ae4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129222025-04-30T06:43:26.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
129232025-04-30T06:43:26.130ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f4f8728-e2b8-421d-ad56-110de110fcf1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129242025-04-30T06:43:26.131ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
129252025-04-30T06:43:26.138ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d6394a2-d3d0-4e42-b9ee-9011d78253a8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129262025-04-30T06:43:26.138ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
129272025-04-30T06:43:26.143ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9aa10cab-9b69-4d6f-a206-2c3e1376e80e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129282025-04-30T06:43:26.145ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
129292025-04-30T06:43:26.155ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd3cb7f8-0996-4b60-b68c-f0ded39b8dcd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129302025-04-30T06:43:26.155ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
129312025-04-30T06:43:26.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7887dae-36c9-4e81-a5bc-947eec1018ce", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129322025-04-30T06:43:26.158ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
129332025-04-30T06:43:26.159ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5e3107b-ed49-46ac-a4df-6b55c8df6d83", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129342025-04-30T06:43:26.159ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
129352025-04-30T06:43:26.161ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d3b2ef2-eb51-444e-b6cd-b3a442873fe6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129362025-04-30T06:43:26.161ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
129372025-04-30T06:43:26.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f88358a-2a59-4e77-9eee-0469f339b1cc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129382025-04-30T06:43:26.162ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
129392025-04-30T06:43:26.165ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebec6b54-0f8f-4fb1-a961-2ea1f43cde69", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129402025-04-30T06:43:26.165ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
129412025-04-30T06:43:26.165ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
129422025-04-30T06:43:26.170ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8f11ec4-e730-49cf-a2b4-5cab4cb1ec2a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129432025-04-30T06:43:26.170ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
129442025-04-30T06:43:26.172ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54fce88b-766c-4165-9c49-09a3ab702b35", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129452025-04-30T06:43:26.173ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
129462025-04-30T06:43:26.174ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97573c42-bccd-4008-878a-9c5815220e53", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129472025-04-30T06:43:26.174ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
129482025-04-30T06:43:26.176ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc987aec-8679-4cd5-9870-2941fd016b69", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129492025-04-30T06:43:26.176ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
129502025-04-30T06:43:26.179ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10008910-2ecf-4205-b2a8-dc56388705df", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129512025-04-30T06:43:26.179ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
129522025-04-30T06:43:26.180ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b187e734-8d7d-49dd-853d-2a8a0b660fd2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129532025-04-30T06:43:26.180ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
129542025-04-30T06:43:26.181ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "335f255a-e38a-4b95-b292-42e95df9186f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129552025-04-30T06:43:26.181ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
129562025-04-30T06:43:26.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2dc5b2b-a493-44c0-87bf-62119a7c5775", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129572025-04-30T06:43:26.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
129582025-04-30T06:43:26.185ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fa508a0-da5e-447a-b9eb-0cfe18421ce4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129592025-04-30T06:43:26.185ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
129602025-04-30T06:43:26.185ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129612025-04-30T06:43:26.186ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129622025-04-30T06:43:26.186ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129632025-04-30T06:43:26.187ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129642025-04-30T06:43:26.198ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129652025-04-30T06:43:26.202ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
129662025-04-30T06:43:26.202ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
129672025-04-30T06:43:26.202ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129682025-04-30T06:43:26.202ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129692025-04-30T06:43:26.202ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129702025-04-30T06:43:26.202ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129712025-04-30T06:43:26.207ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129722025-04-30T06:43:26.210ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
129732025-04-30T06:43:26.211ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129742025-04-30T06:43:26.211ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129752025-04-30T06:43:26.211ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129762025-04-30T06:43:26.217ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129772025-04-30T06:43:26.217ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129782025-04-30T06:43:26.217ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129792025-04-30T06:43:26.217ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
129802025-04-30T06:43:26.219ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129812025-04-30T06:43:26.301ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
129822025-04-30T06:43:26.301ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
129832025-04-30T06:43:26.302ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:49893 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
129842025-04-30T06:43:26.302ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:49893 req_id = 0962901c-50ee-43eb-9cda-dd6f325d7a39 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
129852025-04-30T06:43:26.303ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 748 local_addr = [::1]:51119 method = GET remote_addr = [::1]:49893 req_id = 0962901c-50ee-43eb-9cda-dd6f325d7a39 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
129862025-04-30T06:43:26.303ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0962901c-50ee-43eb-9cda-dd6f325d7a39", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
129872025-04-30T06:43:26.307ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
129882025-04-30T06:43:26.307ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
129892025-04-30T06:43:26.307ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
129902025-04-30T06:43:26.310ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
129912025-04-30T06:43:26.311ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129922025-04-30T06:43:26.311ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129932025-04-30T06:43:26.311ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129942025-04-30T06:43:26.312ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129952025-04-30T06:43:26.313ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129962025-04-30T06:43:26.319ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129972025-04-30T06:43:26.319ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129982025-04-30T06:43:26.319ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
129992025-04-30T06:43:26.319ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130002025-04-30T06:43:26.331ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130012025-04-30T06:43:26.337ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
130022025-04-30T06:43:26.337ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130032025-04-30T06:43:26.337ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130042025-04-30T06:43:26.338ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130052025-04-30T06:43:26.353ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
130062025-04-30T06:43:26.354ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130072025-04-30T06:43:26.354ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
130082025-04-30T06:43:26.355ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130092025-04-30T06:43:26.356ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ac af 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49610 req_id = 275e9418-a7bd-4464-ba39-76c4c724319a
130102025-04-30T06:43:26.356ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44207,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49610 req_id = 275e9418-a7bd-4464-ba39-76c4c724319a
130112025-04-30T06:43:26.356ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
130122025-04-30T06:43:26.356ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130132025-04-30T06:43:26.356ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49610 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 275e9418-a7bd-4464-ba39-76c4c724319a
130142025-04-30T06:43:26.358ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4271, tv_nsec: 157514666 } })
130152025-04-30T06:43:26.358ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
130162025-04-30T06:43:26.358ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6d 1b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e m...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55249 req_id = 091bab82-16ca-4e58-b6a2-ce01647ec069
130172025-04-30T06:43:26.359ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27931,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55249 req_id = 091bab82-16ca-4e58-b6a2-ce01647ec069
130182025-04-30T06:43:26.359ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130192025-04-30T06:43:26.359ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55249 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 091bab82-16ca-4e58-b6a2-ce01647ec069
130202025-04-30T06:43:26.359ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
130212025-04-30T06:43:26.426ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
130222025-04-30T06:43:26.426ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
130232025-04-30T06:43:26.426ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
130242025-04-30T06:43:26.426ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
130252025-04-30T06:43:26.434ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
130262025-04-30T06:43:26.442ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
130272025-04-30T06:43:26.443ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c3 be 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63676 req_id = 10c3720d-02c3-4c7b-94ae-c753612a219b
130282025-04-30T06:43:26.443ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50110,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63676 req_id = 10c3720d-02c3-4c7b-94ae-c753612a219b
130292025-04-30T06:43:26.444ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
130302025-04-30T06:43:26.444ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130312025-04-30T06:43:26.444ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63676 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 10c3720d-02c3-4c7b-94ae-c753612a219b
130322025-04-30T06:43:26.444ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4271, tv_nsec: 243802129 } })
130332025-04-30T06:43:26.445ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
130342025-04-30T06:43:26.445ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a2 46 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .F..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52025 req_id = cce64939-12e6-4f1c-ac29-ef93e1cb5fe7
130352025-04-30T06:43:26.445ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41542,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52025 req_id = cce64939-12e6-4f1c-ac29-ef93e1cb5fe7
130362025-04-30T06:43:26.445ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130372025-04-30T06:43:26.445ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52025 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cce64939-12e6-4f1c-ac29-ef93e1cb5fe7
130382025-04-30T06:43:26.446ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
130392025-04-30T06:43:26.551ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
130402025-04-30T06:43:26.552ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
130412025-04-30T06:43:26.553ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
130422025-04-30T06:43:26.553ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
130432025-04-30T06:43:26.570ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
130442025-04-30T06:43:26.595ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
130452025-04-30T06:43:26.596ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
130462025-04-30T06:43:26.596ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
130472025-04-30T06:43:26.596ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 722.721326ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
130482025-04-30T06:43:26.596ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
130492025-04-30T06:43:26.596ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
130502025-04-30T06:43:26.596ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130512025-04-30T06:43:26.596ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130522025-04-30T06:43:26.597ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130532025-04-30T06:43:26.598ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130542025-04-30T06:43:26.599ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 36 df 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 6............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52639 req_id = b38a238e-a2b7-447f-b4a9-4c86e4ab3aa3
130552025-04-30T06:43:26.600ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14047,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52639 req_id = b38a238e-a2b7-447f-b4a9-4c86e4ab3aa3
130562025-04-30T06:43:26.600ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
130572025-04-30T06:43:26.600ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130582025-04-30T06:43:26.600ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52639 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b38a238e-a2b7-447f-b4a9-4c86e4ab3aa3
130592025-04-30T06:43:26.601ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4271, tv_nsec: 400419378 } })
130602025-04-30T06:43:26.601ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
130612025-04-30T06:43:26.601ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 21 9f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e !...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53176 req_id = e999e087-8715-4c4f-8286-e62e70d2be9c
130622025-04-30T06:43:26.601ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8607,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53176 req_id = e999e087-8715-4c4f-8286-e62e70d2be9c
130632025-04-30T06:43:26.602ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130642025-04-30T06:43:26.602ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53176 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e999e087-8715-4c4f-8286-e62e70d2be9c
130652025-04-30T06:43:26.602ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130662025-04-30T06:43:26.607ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
130672025-04-30T06:43:26.607ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130682025-04-30T06:43:26.607ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130692025-04-30T06:43:26.607ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130702025-04-30T06:43:26.610ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
130712025-04-30T06:43:26.680ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
130722025-04-30T06:43:26.680ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
130732025-04-30T06:43:26.681ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
130742025-04-30T06:43:26.681ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
130752025-04-30T06:43:26.683ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
130762025-04-30T06:43:26.684ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
130772025-04-30T06:43:26.684ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
130782025-04-30T06:43:26.684ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
130792025-04-30T06:43:26.685ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
130802025-04-30T06:43:26.686ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
130812025-04-30T06:43:26.686ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
130822025-04-30T06:43:26.686ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
130832025-04-30T06:43:26.687ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ac 4a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .J..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53875 req_id = 7dbc0392-9f00-47b3-8d10-5d5f3ce7e2cd
130842025-04-30T06:43:26.687ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44106,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53875 req_id = 7dbc0392-9f00-47b3-8d10-5d5f3ce7e2cd
130852025-04-30T06:43:26.687ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
130862025-04-30T06:43:26.687ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130872025-04-30T06:43:26.687ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53875 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7dbc0392-9f00-47b3-8d10-5d5f3ce7e2cd
130882025-04-30T06:43:26.688ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4271, tv_nsec: 487048822 } })
130892025-04-30T06:43:26.688ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
130902025-04-30T06:43:26.688ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: df 2b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .+..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52107 req_id = 4e875f18-81f2-4da6-b72e-8a601b5dd134
130912025-04-30T06:43:26.688ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57131,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52107 req_id = 4e875f18-81f2-4da6-b72e-8a601b5dd134
130922025-04-30T06:43:26.689ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130932025-04-30T06:43:26.689ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52107 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4e875f18-81f2-4da6-b72e-8a601b5dd134
130942025-04-30T06:43:26.759ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
130952025-04-30T06:43:26.760ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130962025-04-30T06:43:26.760ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130972025-04-30T06:43:26.760ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130982025-04-30T06:43:26.760ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
130992025-04-30T06:43:26.761ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131002025-04-30T06:43:26.761ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
131012025-04-30T06:43:26.761ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
131022025-04-30T06:43:26.761ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131032025-04-30T06:43:26.762ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131042025-04-30T06:43:26.764ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9c d3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54464 req_id = 4a36b4ae-ca51-47cf-881f-22eb88ed7fd5
131052025-04-30T06:43:26.764ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40147,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54464 req_id = 4a36b4ae-ca51-47cf-881f-22eb88ed7fd5
131062025-04-30T06:43:26.764ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
131072025-04-30T06:43:26.764ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40147, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54464 req_id = 4a36b4ae-ca51-47cf-881f-22eb88ed7fd5
131082025-04-30T06:43:26.765ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
131092025-04-30T06:43:26.766ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
131102025-04-30T06:43:26.766ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
131112025-04-30T06:43:26.766ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131122025-04-30T06:43:26.767ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131132025-04-30T06:43:26.767ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131142025-04-30T06:43:26.767ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131152025-04-30T06:43:26.768ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131162025-04-30T06:43:26.773ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131172025-04-30T06:43:26.773ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131182025-04-30T06:43:26.773ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
131192025-04-30T06:43:26.773ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131202025-04-30T06:43:26.774ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
131212025-04-30T06:43:26.774ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
131222025-04-30T06:43:26.776ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "21b6adaf-0f2d-4082-a9cc-d20d3fdbe158", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131232025-04-30T06:43:26.776ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
131242025-04-30T06:43:26.776ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131252025-04-30T06:43:26.776ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
131262025-04-30T06:43:26.777ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131272025-04-30T06:43:26.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d84cca7-3314-4fbb-af40-35e3ffaaa12e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131282025-04-30T06:43:26.780ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
131292025-04-30T06:43:26.782ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00937a9c-37e1-435a-958f-6e2eaac24bfc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131302025-04-30T06:43:26.782ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
131312025-04-30T06:43:26.784ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d60d9b3c-345b-4cfc-a070-da9ec7c3f023", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131322025-04-30T06:43:26.784ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
131332025-04-30T06:43:26.785ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
131342025-04-30T06:43:26.787ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "982a8748-428d-4ba6-a9fd-78629df411a6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131352025-04-30T06:43:26.787ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
131362025-04-30T06:43:26.790ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ba54edf-2fba-42b3-996b-17401e80bccc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131372025-04-30T06:43:26.790ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
131382025-04-30T06:43:26.792ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0785027d-d0a6-4de7-8c5f-72cc7c107e1e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131392025-04-30T06:43:26.792ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
131402025-04-30T06:43:26.794ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aede1d8c-5eee-4311-81b4-0b53ce7ffd0d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131412025-04-30T06:43:26.794ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
131422025-04-30T06:43:26.795ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5832aef0-8be8-45dd-9df2-e14513d5ca80", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131432025-04-30T06:43:26.795ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
131442025-04-30T06:43:26.796ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "814937f0-ac8c-439b-b06c-e949f4cf0856", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131452025-04-30T06:43:26.797ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
131462025-04-30T06:43:26.798ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c8ba14e-5da8-41f1-8f98-2c73d906bf4c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131472025-04-30T06:43:26.798ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
131482025-04-30T06:43:26.800ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f77286ff-24bb-4d48-b4a4-2507ffb88c42", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131492025-04-30T06:43:26.800ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
131502025-04-30T06:43:26.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec9123c-a931-4b1b-89ea-f2467e8c8426", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131512025-04-30T06:43:26.802ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
131522025-04-30T06:43:26.805ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a85d65f3-ce4f-433f-ae5f-0c0ee0458752", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131532025-04-30T06:43:26.806ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
131542025-04-30T06:43:26.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cad7e1f-011d-47e6-9f2b-4c0ad1db2b03", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131552025-04-30T06:43:26.809ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
131562025-04-30T06:43:26.810ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
131572025-04-30T06:43:26.813ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e7ec2a5-bf93-4795-ac2c-1eb2eb2beacd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131582025-04-30T06:43:26.813ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
131592025-04-30T06:43:26.816ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b3e398c-dfa2-40f5-aa5c-7c0c34f02970", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131602025-04-30T06:43:26.817ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
131612025-04-30T06:43:26.819ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae85f4f6-d3b8-4c7a-96fd-71cae6603389", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131622025-04-30T06:43:26.819ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
131632025-04-30T06:43:26.824ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb5683bf-80cf-491c-98e9-61a9433b6754", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131642025-04-30T06:43:26.825ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
131652025-04-30T06:43:26.826ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80f30782-c3d8-4574-bcf4-ad8c6452d1e4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131662025-04-30T06:43:26.827ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
131672025-04-30T06:43:26.828ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bae4598a-db64-4626-88ff-6689d97e6cc6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131682025-04-30T06:43:26.829ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
131692025-04-30T06:43:26.830ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f9d2e7e-c6d8-4aac-95bd-4c24b547af88", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131702025-04-30T06:43:26.830ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
131712025-04-30T06:43:26.832ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c405e50-0001-4bf3-8ed7-acf010534c6a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131722025-04-30T06:43:26.832ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
131732025-04-30T06:43:26.834ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1026b8f-7323-44d3-aad7-cc9d57730adf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131742025-04-30T06:43:26.834ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
131752025-04-30T06:43:26.837ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19398c8b-43c4-45d2-816a-0460265cbac0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131762025-04-30T06:43:26.837ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
131772025-04-30T06:43:26.840ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21c38330-2a1f-4ce7-8b88-3179fc4f98ba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131782025-04-30T06:43:26.840ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
131792025-04-30T06:43:26.843ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d734d938-4880-405e-805b-d692f47118b0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131802025-04-30T06:43:26.843ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
131812025-04-30T06:43:26.850ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d036c2e-11a4-4ae0-a0f0-3d5eafd0e8fd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131822025-04-30T06:43:26.850ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
131832025-04-30T06:43:26.852ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0525e852-373f-4e6a-867c-c9f61b618601", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131842025-04-30T06:43:26.853ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
131852025-04-30T06:43:26.862ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131862025-04-30T06:43:26.862ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131872025-04-30T06:43:26.862ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131882025-04-30T06:43:26.862ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
131892025-04-30T06:43:26.864ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131902025-04-30T06:43:26.865ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "766b2d60-312e-4860-a104-04a983c0160d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131912025-04-30T06:43:26.865ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
131922025-04-30T06:43:26.867ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "671b385a-11cd-4c9b-b0e9-5c7f0cb2c2a2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
131932025-04-30T06:43:26.868ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
131942025-04-30T06:43:26.868ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131952025-04-30T06:43:26.878ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131962025-04-30T06:43:26.878ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131972025-04-30T06:43:26.878ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
131982025-04-30T06:43:26.879ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131992025-04-30T06:43:26.883ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132002025-04-30T06:43:26.883ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
132012025-04-30T06:43:26.884ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132022025-04-30T06:43:26.885ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a8fb148-eb25-4dca-aa1b-7510eee560d2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132032025-04-30T06:43:26.885ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
132042025-04-30T06:43:26.889ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bb841d7-46ab-480c-bb2c-b1ef02f21d7b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132052025-04-30T06:43:26.891ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
132062025-04-30T06:43:26.896ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01ac4987-6fb8-430b-8d9c-a7ddc94f905f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132072025-04-30T06:43:26.899ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
132082025-04-30T06:43:26.903ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb98d734-114b-4ff5-8b62-a2fbd8f3180d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132092025-04-30T06:43:26.903ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
132102025-04-30T06:43:26.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05a88b84-5a52-443f-ad6b-be83ac98fbb7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132112025-04-30T06:43:26.908ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
132122025-04-30T06:43:26.910ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b6211ba-8966-4ae6-b02e-052c7eef4011", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132132025-04-30T06:43:26.910ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
132142025-04-30T06:43:26.914ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea58af8b-3f5d-4c5b-89f1-b860a5969e04", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132152025-04-30T06:43:26.914ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
132162025-04-30T06:43:26.923ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
132172025-04-30T06:43:26.923ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bbbf0a7-bc9e-4d97-868a-cb172a1b5651", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132182025-04-30T06:43:26.924ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
132192025-04-30T06:43:26.929ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28ebd2fd-9592-40b8-aa48-9bb701c11a33", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132202025-04-30T06:43:26.929ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
132212025-04-30T06:43:26.935ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6949288-9d23-4592-98b6-09de91175958", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132222025-04-30T06:43:26.943ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
132232025-04-30T06:43:26.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0e2a3d4-e0f4-49bf-a165-e064400f1003", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132242025-04-30T06:43:26.946ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
132252025-04-30T06:43:26.947ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4eb45d0-54dd-483a-9e31-5bdce539ddc7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132262025-04-30T06:43:26.947ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
132272025-04-30T06:43:26.949ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "267c9f3b-98b3-4325-bb46-4ef5b29e554b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132282025-04-30T06:43:26.949ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
132292025-04-30T06:43:26.950ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a33873ab-0345-4301-a73e-ba66e967b9ad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
132302025-04-30T06:43:26.950ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
132312025-04-30T06:43:26.950ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132322025-04-30T06:43:26.950ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132332025-04-30T06:43:26.950ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132342025-04-30T06:43:26.951ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132352025-04-30T06:43:26.959ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132362025-04-30T06:43:26.968ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
132372025-04-30T06:43:26.968ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
132382025-04-30T06:43:26.968ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132392025-04-30T06:43:26.968ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132402025-04-30T06:43:26.969ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132412025-04-30T06:43:26.969ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132422025-04-30T06:43:26.972ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132432025-04-30T06:43:26.979ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
132442025-04-30T06:43:26.979ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132452025-04-30T06:43:26.979ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132462025-04-30T06:43:26.979ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132472025-04-30T06:43:26.980ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132482025-04-30T06:43:26.980ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132492025-04-30T06:43:26.980ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132502025-04-30T06:43:26.980ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
132512025-04-30T06:43:26.981ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132522025-04-30T06:43:27.058ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
132532025-04-30T06:43:27.058ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
132542025-04-30T06:43:27.058ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:41067 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
132552025-04-30T06:43:27.062ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:41067 req_id = 4876282e-a70d-4a2d-bd7d-c595bbd64abb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
132562025-04-30T06:43:27.063ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 687 local_addr = [::1]:51119 method = GET remote_addr = [::1]:41067 req_id = 4876282e-a70d-4a2d-bd7d-c595bbd64abb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
132572025-04-30T06:43:27.064ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132582025-04-30T06:43:27.077ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132592025-04-30T06:43:27.077ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132602025-04-30T06:43:27.077ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
132612025-04-30T06:43:27.078ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132622025-04-30T06:43:27.078ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4876282e-a70d-4a2d-bd7d-c595bbd64abb", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
132632025-04-30T06:43:27.081ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
132642025-04-30T06:43:27.081ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
132652025-04-30T06:43:27.081ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
132662025-04-30T06:43:27.084ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
132672025-04-30T06:43:27.084ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132682025-04-30T06:43:27.084ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132692025-04-30T06:43:27.084ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132702025-04-30T06:43:27.084ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132712025-04-30T06:43:27.091ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132722025-04-30T06:43:27.091ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
132732025-04-30T06:43:27.093ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132742025-04-30T06:43:27.093ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132752025-04-30T06:43:27.099ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
132762025-04-30T06:43:27.100ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132772025-04-30T06:43:27.100ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132782025-04-30T06:43:27.100ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132792025-04-30T06:43:27.152ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
132802025-04-30T06:43:27.152ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
132812025-04-30T06:43:27.152ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 556.227628ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
132822025-04-30T06:43:27.153ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
132832025-04-30T06:43:27.153ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
132842025-04-30T06:43:27.153ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132852025-04-30T06:43:27.153ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132862025-04-30T06:43:27.153ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132872025-04-30T06:43:27.154ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132882025-04-30T06:43:27.165ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132892025-04-30T06:43:27.177ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
132902025-04-30T06:43:27.177ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132912025-04-30T06:43:27.177ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132922025-04-30T06:43:27.177ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132932025-04-30T06:43:27.185ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
132942025-04-30T06:43:27.186ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
132952025-04-30T06:43:27.187ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
132962025-04-30T06:43:27.189ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f9 fb 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54907 req_id = 71398e1d-2fac-438d-a58d-cf4736ab0aab
132972025-04-30T06:43:27.190ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63995,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54907 req_id = 71398e1d-2fac-438d-a58d-cf4736ab0aab
132982025-04-30T06:43:27.190ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
132992025-04-30T06:43:27.190ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133002025-04-30T06:43:27.191ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54907 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 71398e1d-2fac-438d-a58d-cf4736ab0aab
133012025-04-30T06:43:27.192ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4271, tv_nsec: 991068959 } })
133022025-04-30T06:43:27.192ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
133032025-04-30T06:43:27.192ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f2 6e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .n..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55806 req_id = fbcbe05b-a399-4434-a1b9-8c36a9c219b6
133042025-04-30T06:43:27.193ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62062,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55806 req_id = fbcbe05b-a399-4434-a1b9-8c36a9c219b6
133052025-04-30T06:43:27.193ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133062025-04-30T06:43:27.193ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55806 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fbcbe05b-a399-4434-a1b9-8c36a9c219b6
133072025-04-30T06:43:27.257ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
133082025-04-30T06:43:27.258ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133092025-04-30T06:43:27.258ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133102025-04-30T06:43:27.258ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133112025-04-30T06:43:27.258ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
133122025-04-30T06:43:27.260ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133132025-04-30T06:43:27.261ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9d b9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65240 req_id = d8ef7240-21ae-498f-864e-648c69213916
133142025-04-30T06:43:27.261ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40377,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65240 req_id = d8ef7240-21ae-498f-864e-648c69213916
133152025-04-30T06:43:27.262ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
133162025-04-30T06:43:27.273ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40377, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65240 req_id = d8ef7240-21ae-498f-864e-648c69213916
133172025-04-30T06:43:27.273ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
133182025-04-30T06:43:27.273ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
133192025-04-30T06:43:27.273ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
133202025-04-30T06:43:27.277ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133212025-04-30T06:43:27.288ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133222025-04-30T06:43:27.288ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133232025-04-30T06:43:27.288ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
133242025-04-30T06:43:27.289ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133252025-04-30T06:43:27.289ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "593b5693-2486-43ed-93fd-e333940f425d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133262025-04-30T06:43:27.290ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
133272025-04-30T06:43:27.293ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0f7e38f-86d1-475d-b18b-96e3dd264de8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133282025-04-30T06:43:27.294ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
133292025-04-30T06:43:27.294ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133302025-04-30T06:43:27.294ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
133312025-04-30T06:43:27.295ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133322025-04-30T06:43:27.301ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9a2800d-5bfe-4e35-9bf8-2f39f11dd615", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133332025-04-30T06:43:27.301ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
133342025-04-30T06:43:27.304ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "557fd088-3887-4ea4-967f-7707011aae2f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133352025-04-30T06:43:27.304ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
133362025-04-30T06:43:27.336ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5fc4c47-a127-493a-8298-79f3f40ca2d9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133372025-04-30T06:43:27.336ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
133382025-04-30T06:43:27.348ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ebef843-4308-4697-9935-f2dec3c32401", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133392025-04-30T06:43:27.349ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
133402025-04-30T06:43:27.353ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f2abac6-e34a-4fdd-86f1-5375ae4e8112", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133412025-04-30T06:43:27.353ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
133422025-04-30T06:43:27.357ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "898449ed-f004-487e-b32c-74f80c59afe9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133432025-04-30T06:43:27.358ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
133442025-04-30T06:43:27.361ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65e013ee-1d41-4d83-89f4-6b2cf9033503", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133452025-04-30T06:43:27.361ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
133462025-04-30T06:43:27.362ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
133472025-04-30T06:43:27.364ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f17a99f-cae2-4e64-a30b-68cf5a1c3f61", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133482025-04-30T06:43:27.364ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
133492025-04-30T06:43:27.365ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7beeaae-6ced-4cc7-8234-2b639bc3df21", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133502025-04-30T06:43:27.366ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
133512025-04-30T06:43:27.367ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a197a31b-cb31-4f2f-a793-105d7741986f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133522025-04-30T06:43:27.368ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
133532025-04-30T06:43:27.371ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "158526d5-b676-430a-bb7b-7a9e622215b3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133542025-04-30T06:43:27.371ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
133552025-04-30T06:43:27.374ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db8d9882-9d93-4200-8081-1259a8a59462", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133562025-04-30T06:43:27.374ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
133572025-04-30T06:43:27.383ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bb2492a-c541-4374-95f2-4dca2f2e4131", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133582025-04-30T06:43:27.389ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
133592025-04-30T06:43:27.392ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b10ff6d5-6605-4a8e-b7cb-4070e8686586", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133602025-04-30T06:43:27.393ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
133612025-04-30T06:43:27.395ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e96008fd-ec6f-4411-b9e3-1dfd2636bc26", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133622025-04-30T06:43:27.395ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
133632025-04-30T06:43:27.398ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5714c924-441b-4dff-8370-3ff75f5eaca3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133642025-04-30T06:43:27.399ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
133652025-04-30T06:43:27.418ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133662025-04-30T06:43:27.418ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133672025-04-30T06:43:27.418ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133682025-04-30T06:43:27.418ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
133692025-04-30T06:43:27.424ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133702025-04-30T06:43:27.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc0cb22c-5f56-4d37-83e9-65e34e82d904", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133712025-04-30T06:43:27.427ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
133722025-04-30T06:43:27.428ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133732025-04-30T06:43:27.441ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133742025-04-30T06:43:27.442ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133752025-04-30T06:43:27.442ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
133762025-04-30T06:43:27.443ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133772025-04-30T06:43:27.447ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94c497ab-6eb2-4782-a8de-f0381be9b726", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133782025-04-30T06:43:27.447ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
133792025-04-30T06:43:27.447ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133802025-04-30T06:43:27.447ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
133812025-04-30T06:43:27.448ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133822025-04-30T06:43:27.452ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fa52acb-b805-4c42-b152-5a8100a78968", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133832025-04-30T06:43:27.452ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
133842025-04-30T06:43:27.454ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c90a1959-1f4d-4fc6-a841-893484282ef0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133852025-04-30T06:43:27.454ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
133862025-04-30T06:43:27.456ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3764d488-47d7-43eb-9ec2-99b90b7f0aa0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133872025-04-30T06:43:27.456ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
133882025-04-30T06:43:27.458ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd0225fe-45ee-4c71-b0df-7e6524024018", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133892025-04-30T06:43:27.458ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
133902025-04-30T06:43:27.477ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a923221a-2ee6-4a1c-8045-1c15c9e21330", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133912025-04-30T06:43:27.477ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
133922025-04-30T06:43:27.491ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66b33a75-57e6-41bf-979d-42a2b59f332a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133932025-04-30T06:43:27.491ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
133942025-04-30T06:43:27.496ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
133952025-04-30T06:43:27.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f43ffe2-8ae2-4d24-8ff2-01b1e1a5bbc9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133962025-04-30T06:43:27.505ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
133972025-04-30T06:43:27.515ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ccec9f0-04e9-487c-b430-2fcbd3fcbc14", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
133982025-04-30T06:43:27.515ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
133992025-04-30T06:43:27.521ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5843122f-0cf1-4f34-bae3-90d8976336b4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134002025-04-30T06:43:27.522ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
134012025-04-30T06:43:27.531ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23c3729d-4c96-4f0b-9362-ba35eb55eaf1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134022025-04-30T06:43:27.531ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
134032025-04-30T06:43:27.533ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5198459-4752-42b4-9628-5656b5209377", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134042025-04-30T06:43:27.533ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
134052025-04-30T06:43:27.534ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a9b0020-985d-4f38-bf64-7d0d4cf8177b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134062025-04-30T06:43:27.534ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
134072025-04-30T06:43:27.535ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d09665f-e88f-42ac-8720-5db2230cd2ef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134082025-04-30T06:43:27.536ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
134092025-04-30T06:43:27.538ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a534d3d-a9ec-4afb-be15-7941d8ce8530", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134102025-04-30T06:43:27.538ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
134112025-04-30T06:43:27.545ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "989d353e-f8d3-4b5b-bb6b-1a95f714db55", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134122025-04-30T06:43:27.547ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
134132025-04-30T06:43:27.547ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134142025-04-30T06:43:27.547ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134152025-04-30T06:43:27.547ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134162025-04-30T06:43:27.547ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
134172025-04-30T06:43:27.549ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134182025-04-30T06:43:27.553ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134192025-04-30T06:43:27.564ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134202025-04-30T06:43:27.564ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134212025-04-30T06:43:27.564ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
134222025-04-30T06:43:27.576ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134232025-04-30T06:43:27.582ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26be150f-1aa9-40fd-a172-1146c3eabf0b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134242025-04-30T06:43:27.583ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
134252025-04-30T06:43:27.583ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134262025-04-30T06:43:27.583ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
134272025-04-30T06:43:27.584ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134282025-04-30T06:43:27.584ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
134292025-04-30T06:43:27.586ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134302025-04-30T06:43:27.589ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 90ef963d-c1e6-4838-b010-4d541bc7b748 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134312025-04-30T06:43:27.590ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 90ef963d-c1e6-4838-b010-4d541bc7b748 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134322025-04-30T06:43:27.590ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 90ef963d-c1e6-4838-b010-4d541bc7b748 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134332025-04-30T06:43:27.590ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 90ef963d-c1e6-4838-b010-4d541bc7b748 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134342025-04-30T06:43:27.595ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a39b9db0-6b3c-4277-9bbf-9a510f4fc56d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134352025-04-30T06:43:27.595ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
134362025-04-30T06:43:27.596ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 7194 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 90ef963d-c1e6-4838-b010-4d541bc7b748 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134372025-04-30T06:43:27.598ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "90ef963d-c1e6-4838-b010-4d541bc7b748", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134382025-04-30T06:43:27.599ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
134392025-04-30T06:43:27.601ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = a6da51e7-0725-4a85-8e49-afed828ddf22 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
134402025-04-30T06:43:27.602ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = a6da51e7-0725-4a85-8e49-afed828ddf22 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
134412025-04-30T06:43:27.602ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = a6da51e7-0725-4a85-8e49-afed828ddf22 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
134422025-04-30T06:43:27.602ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = a6da51e7-0725-4a85-8e49-afed828ddf22 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
134432025-04-30T06:43:27.606ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 4883 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = a6da51e7-0725-4a85-8e49-afed828ddf22 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
134442025-04-30T06:43:27.607ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6da51e7-0725-4a85-8e49-afed828ddf22", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134452025-04-30T06:43:27.607ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
134462025-04-30T06:43:27.607ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
134472025-04-30T06:43:27.607ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
134482025-04-30T06:43:27.609ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34f0d331-be17-4f14-8c6e-805ac98cfc53", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134492025-04-30T06:43:27.609ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
134502025-04-30T06:43:27.622ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31bf4b01-c8d0-4103-b10c-9757d55cc4c0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134512025-04-30T06:43:27.622ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
134522025-04-30T06:43:27.634ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
134532025-04-30T06:43:27.634ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5276ba8-76cd-426c-8c8f-225408b5873d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134542025-04-30T06:43:27.636ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
134552025-04-30T06:43:27.643ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b89c3821-f7ab-4221-996f-89a1f8d7b37b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134562025-04-30T06:43:27.644ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
134572025-04-30T06:43:27.645ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5ba188d-4040-432d-a5db-cf2ce31d5689", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134582025-04-30T06:43:27.645ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
134592025-04-30T06:43:27.647ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05ecba3b-5853-4f57-aa0b-d036e11f88ac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134602025-04-30T06:43:27.647ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
134612025-04-30T06:43:27.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edfc96c5-00c0-48aa-8fe7-b5cb142149a6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134622025-04-30T06:43:27.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
134632025-04-30T06:43:27.650ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a81d5b3a-8048-4c4b-b950-ec775f3f7133", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
134642025-04-30T06:43:27.650ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134652025-04-30T06:43:27.650ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134662025-04-30T06:43:27.650ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134672025-04-30T06:43:27.650ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134682025-04-30T06:43:27.652ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134692025-04-30T06:43:27.662ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134702025-04-30T06:43:27.674ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134712025-04-30T06:43:27.674ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134722025-04-30T06:43:27.674ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134732025-04-30T06:43:27.674ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134742025-04-30T06:43:27.674ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134752025-04-30T06:43:27.675ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134762025-04-30T06:43:27.681ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134772025-04-30T06:43:27.702ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134782025-04-30T06:43:27.702ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134792025-04-30T06:43:27.702ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134802025-04-30T06:43:27.703ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134812025-04-30T06:43:27.704ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134822025-04-30T06:43:27.704ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134832025-04-30T06:43:27.704ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134842025-04-30T06:43:27.704ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
134852025-04-30T06:43:27.708ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134862025-04-30T06:43:27.708ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
134872025-04-30T06:43:27.708ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
134882025-04-30T06:43:27.708ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
134892025-04-30T06:43:27.717ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
134902025-04-30T06:43:27.814ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
134912025-04-30T06:43:27.814ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
134922025-04-30T06:43:27.814ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:42045 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
134932025-04-30T06:43:27.815ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
134942025-04-30T06:43:27.815ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
134952025-04-30T06:43:27.815ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
134962025-04-30T06:43:27.816ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
134972025-04-30T06:43:27.816ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
134982025-04-30T06:43:27.817ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:42045 req_id = 41765a41-20ad-4279-90b3-fd341992d181 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
134992025-04-30T06:43:27.828ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 10676 local_addr = [::1]:51119 method = GET remote_addr = [::1]:42045 req_id = 41765a41-20ad-4279-90b3-fd341992d181 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
135002025-04-30T06:43:27.829ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135012025-04-30T06:43:27.842ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135022025-04-30T06:43:27.842ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135032025-04-30T06:43:27.842ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135042025-04-30T06:43:27.843ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135052025-04-30T06:43:27.846ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
135062025-04-30T06:43:27.846ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
135072025-04-30T06:43:27.847ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135082025-04-30T06:43:27.848ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135092025-04-30T06:43:27.848ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135102025-04-30T06:43:27.848ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135112025-04-30T06:43:27.849ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "41765a41-20ad-4279-90b3-fd341992d181", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
135122025-04-30T06:43:27.853ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135132025-04-30T06:43:27.853ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135142025-04-30T06:43:27.853ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
135152025-04-30T06:43:27.856ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135162025-04-30T06:43:27.857ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135172025-04-30T06:43:27.857ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135182025-04-30T06:43:27.857ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135192025-04-30T06:43:27.861ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135202025-04-30T06:43:27.865ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135212025-04-30T06:43:27.866ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135222025-04-30T06:43:27.867ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135232025-04-30T06:43:27.867ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135242025-04-30T06:43:27.880ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
135252025-04-30T06:43:27.880ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135262025-04-30T06:43:27.880ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135272025-04-30T06:43:27.881ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135282025-04-30T06:43:27.935ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
135292025-04-30T06:43:27.936ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
135302025-04-30T06:43:27.936ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 783.051819ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
135312025-04-30T06:43:27.936ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
135322025-04-30T06:43:27.937ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135332025-04-30T06:43:27.937ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135342025-04-30T06:43:27.938ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135352025-04-30T06:43:27.938ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135362025-04-30T06:43:27.940ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135372025-04-30T06:43:27.944ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
135382025-04-30T06:43:27.948ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135392025-04-30T06:43:27.954ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
135402025-04-30T06:43:27.954ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135412025-04-30T06:43:27.955ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135422025-04-30T06:43:27.955ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135432025-04-30T06:43:27.958ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
135442025-04-30T06:43:27.958ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
135452025-04-30T06:43:27.959ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a2 2a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .*..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59273 req_id = 7588e8a1-cecd-421e-b7a5-8353ce09da56
135462025-04-30T06:43:27.959ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41514,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59273 req_id = 7588e8a1-cecd-421e-b7a5-8353ce09da56
135472025-04-30T06:43:27.959ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
135482025-04-30T06:43:27.960ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
135492025-04-30T06:43:27.960ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59273 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7588e8a1-cecd-421e-b7a5-8353ce09da56
135502025-04-30T06:43:27.960ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4272, tv_nsec: 759337534 } })
135512025-04-30T06:43:27.960ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
135522025-04-30T06:43:27.960ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3c 02 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e <...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52693 req_id = 6d0280ed-059d-4773-9f65-93f34addded3
135532025-04-30T06:43:27.960ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15362,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52693 req_id = 6d0280ed-059d-4773-9f65-93f34addded3
135542025-04-30T06:43:27.960ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
135552025-04-30T06:43:27.961ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52693 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6d0280ed-059d-4773-9f65-93f34addded3
135562025-04-30T06:43:28.041ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
135572025-04-30T06:43:28.042ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135582025-04-30T06:43:28.042ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135592025-04-30T06:43:28.042ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135602025-04-30T06:43:28.042ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135612025-04-30T06:43:28.043ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135622025-04-30T06:43:28.046ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:65396 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
135632025-04-30T06:43:28.046ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 55 b9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c U............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65369 req_id = 600d6cde-214e-43c4-9214-eeff8275f317
135642025-04-30T06:43:28.046ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21945,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65369 req_id = 600d6cde-214e-43c4-9214-eeff8275f317
135652025-04-30T06:43:28.046ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
135662025-04-30T06:43:28.046ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21945, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65369 req_id = 600d6cde-214e-43c4-9214-eeff8275f317
135672025-04-30T06:43:28.047ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
135682025-04-30T06:43:28.047ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
135692025-04-30T06:43:28.047ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
135702025-04-30T06:43:28.048ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:65396 req_id = bc1df342-33e8-40b5-b27a-adad1e44da24 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
135712025-04-30T06:43:28.048ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 505 local_addr = [::1]:54156 method = GET remote_addr = [::1]:65396 req_id = bc1df342-33e8-40b5-b27a-adad1e44da24 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
135722025-04-30T06:43:28.048ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135732025-04-30T06:43:28.060ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135742025-04-30T06:43:28.060ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135752025-04-30T06:43:28.061ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135762025-04-30T06:43:28.062ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135772025-04-30T06:43:28.087ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135782025-04-30T06:43:28.088ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135792025-04-30T06:43:28.089ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135802025-04-30T06:43:28.102ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f34609dd-1180-48a9-aea7-247fe0cfab94", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
135812025-04-30T06:43:28.104ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
135822025-04-30T06:43:28.138ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d16d3efc-a9eb-42fb-bb61-00c1b44d7514", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
135832025-04-30T06:43:28.142ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
135842025-04-30T06:43:28.167ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f0aa414-8980-40fe-8402-d32a30e0bec0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
135852025-04-30T06:43:28.167ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
135862025-04-30T06:43:28.169ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b97efa2c-90a1-4845-82e1-56a5304d8f3a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
135872025-04-30T06:43:28.170ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
135882025-04-30T06:43:28.172ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cd472da-bfda-40ae-ad6f-e93d40300c84", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
135892025-04-30T06:43:28.172ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
135902025-04-30T06:43:28.177ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b13fdd3-e218-405d-8ef5-4e1b48edd707", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
135912025-04-30T06:43:28.177ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
135922025-04-30T06:43:28.178ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
135932025-04-30T06:43:28.182ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14b15613-e1b0-4f9e-a926-fce6880a8a53", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
135942025-04-30T06:43:28.190ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
135952025-04-30T06:43:28.193ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41de2ecd-65b7-4b10-8e8d-5ea216c20517", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
135962025-04-30T06:43:28.194ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
135972025-04-30T06:43:28.198ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb064d1b-963c-4afc-9453-d142de9b895b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
135982025-04-30T06:43:28.198ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
135992025-04-30T06:43:28.199ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b3e45fb-4908-42d9-9fce-5419626faff3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136002025-04-30T06:43:28.199ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
136012025-04-30T06:43:28.202ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19dd0b1a-be24-40f3-aca5-4019a17fda80", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136022025-04-30T06:43:28.202ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
136032025-04-30T06:43:28.206ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bdc2821-8f86-4988-9c0e-deaf9b6f287a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136042025-04-30T06:43:28.206ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
136052025-04-30T06:43:28.208ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3160ddea-bedd-4211-8a00-40aa53612e64", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136062025-04-30T06:43:28.209ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
136072025-04-30T06:43:28.211ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "778b394a-1f82-4164-9d8d-0fe7cdbff510", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136082025-04-30T06:43:28.211ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
136092025-04-30T06:43:28.213ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21f562c8-32a9-4de9-8bf5-23d2b457f79c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136102025-04-30T06:43:28.220ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
136112025-04-30T06:43:28.222ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0e07e47-c0b3-40f7-a4ad-f55aefa11841", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136122025-04-30T06:43:28.222ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
136132025-04-30T06:43:28.225ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28c436e2-6d7e-4974-b59d-7a8c7e6cb289", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136142025-04-30T06:43:28.226ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
136152025-04-30T06:43:28.229ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136162025-04-30T06:43:28.229ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136172025-04-30T06:43:28.229ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136182025-04-30T06:43:28.229ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
136192025-04-30T06:43:28.230ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136202025-04-30T06:43:28.230ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c9e1ac9-2ce8-4fb8-a63c-4943eba91359", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136212025-04-30T06:43:28.231ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
136222025-04-30T06:43:28.235ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136232025-04-30T06:43:28.244ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136242025-04-30T06:43:28.244ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136252025-04-30T06:43:28.244ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
136262025-04-30T06:43:28.245ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136272025-04-30T06:43:28.247ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51c71574-a488-4a42-90ad-964ca5cf95a4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136282025-04-30T06:43:28.247ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
136292025-04-30T06:43:28.248ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136302025-04-30T06:43:28.249ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
136312025-04-30T06:43:28.251ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136322025-04-30T06:43:28.252ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47b23e57-da98-427c-ab48-5ccdba5e1542", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136332025-04-30T06:43:28.253ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
136342025-04-30T06:43:28.256ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed8cce99-9d28-414e-b0b8-f65e127b1142", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136352025-04-30T06:43:28.257ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
136362025-04-30T06:43:28.260ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5327684-5412-41fd-9165-47813fadff8c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136372025-04-30T06:43:28.260ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
136382025-04-30T06:43:28.263ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2be6dbcd-5393-4355-a716-d06e933e51e7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136392025-04-30T06:43:28.266ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
136402025-04-30T06:43:28.269ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23613ac6-5c2c-40ab-afd8-ab4feb949783", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136412025-04-30T06:43:28.269ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
136422025-04-30T06:43:28.272ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ca894f4-0cd6-4d3e-a97f-163f382971ad", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136432025-04-30T06:43:28.273ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
136442025-04-30T06:43:28.281ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d5d1838-e784-46cc-9f85-eac50415983c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136452025-04-30T06:43:28.281ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
136462025-04-30T06:43:28.286ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d12a1a8-9056-43ba-88a6-3c9ece080137", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136472025-04-30T06:43:28.286ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
136482025-04-30T06:43:28.289ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
136492025-04-30T06:43:28.291ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94058103-1d3d-4251-905d-df582a05be6e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136502025-04-30T06:43:28.291ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
136512025-04-30T06:43:28.295ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de7e96fd-b8b2-4411-906b-c0b80d103559", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136522025-04-30T06:43:28.295ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
136532025-04-30T06:43:28.300ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcab658b-b454-4b84-8771-e9b30ffb53c6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136542025-04-30T06:43:28.301ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
136552025-04-30T06:43:28.303ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4902480-69a8-4abc-8687-6cf0b8ce0497", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136562025-04-30T06:43:28.303ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
136572025-04-30T06:43:28.305ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b2f35e-4cba-498b-a7fa-2485e4fa2523", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136582025-04-30T06:43:28.305ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
136592025-04-30T06:43:28.307ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c86474e3-2a12-40e1-9b01-1b862a2c55a5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136602025-04-30T06:43:28.307ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
136612025-04-30T06:43:28.308ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "baac165f-1490-423a-ac9f-98fbd1d85c21", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136622025-04-30T06:43:28.308ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
136632025-04-30T06:43:28.309ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4f8dc54-697b-4a30-841e-0e924bbf11b4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136642025-04-30T06:43:28.309ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
136652025-04-30T06:43:28.312ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "235d7a47-f8cf-49e1-8843-d0885ae5594e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136662025-04-30T06:43:28.312ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
136672025-04-30T06:43:28.314ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "116e3a95-c136-4b3a-a0eb-4f849881a685", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136682025-04-30T06:43:28.314ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
136692025-04-30T06:43:28.317ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96fdb8f0-3298-4fb6-811a-e4a88534bf20", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136702025-04-30T06:43:28.318ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
136712025-04-30T06:43:28.320ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fd61955-bf32-47d4-ab67-339876d9eace", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136722025-04-30T06:43:28.320ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
136732025-04-30T06:43:28.325ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0c74e9f-2604-4fca-8426-a2318a994fa5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136742025-04-30T06:43:28.325ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
136752025-04-30T06:43:28.328ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e48bfe3b-9af9-420c-b90d-dbd61eb41620", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136762025-04-30T06:43:28.328ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
136772025-04-30T06:43:28.330ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd670825-b64e-4ff6-b29b-401c3921d63c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136782025-04-30T06:43:28.330ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
136792025-04-30T06:43:28.332ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fd7e1ed-3c25-4a3f-a363-226c5261cb28", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136802025-04-30T06:43:28.332ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
136812025-04-30T06:43:28.333ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e34e3bdb-720a-4371-b311-e9d87e8f2edd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136822025-04-30T06:43:28.333ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
136832025-04-30T06:43:28.334ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74a9ec1a-199b-464a-bdbe-abafd61b48b7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
136842025-04-30T06:43:28.334ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136852025-04-30T06:43:28.334ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136862025-04-30T06:43:28.334ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136872025-04-30T06:43:28.334ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136882025-04-30T06:43:28.335ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136892025-04-30T06:43:28.338ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136902025-04-30T06:43:28.344ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136912025-04-30T06:43:28.344ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136922025-04-30T06:43:28.344ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136932025-04-30T06:43:28.344ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136942025-04-30T06:43:28.344ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136952025-04-30T06:43:28.344ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136962025-04-30T06:43:28.346ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136972025-04-30T06:43:28.346ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136982025-04-30T06:43:28.346ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136992025-04-30T06:43:28.346ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
137002025-04-30T06:43:28.346ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137012025-04-30T06:43:28.355ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137022025-04-30T06:43:28.358ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137032025-04-30T06:43:28.358ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137042025-04-30T06:43:28.359ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137052025-04-30T06:43:28.359ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137062025-04-30T06:43:28.364ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137072025-04-30T06:43:28.369ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137082025-04-30T06:43:28.374ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137092025-04-30T06:43:28.374ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
137102025-04-30T06:43:28.377ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137112025-04-30T06:43:28.470ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
137122025-04-30T06:43:28.470ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
137132025-04-30T06:43:28.471ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:47152 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137142025-04-30T06:43:28.474ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
137152025-04-30T06:43:28.476ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137162025-04-30T06:43:28.476ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
137172025-04-30T06:43:28.478ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137182025-04-30T06:43:28.478ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:47152 req_id = 9c549d32-655e-4fa2-97b4-ef850a0dd6b0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
137192025-04-30T06:43:29.053ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 574292 local_addr = [::1]:51119 method = GET remote_addr = [::1]:47152 req_id = 9c549d32-655e-4fa2-97b4-ef850a0dd6b0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
137202025-04-30T06:43:29.053ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:43372 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137212025-04-30T06:43:29.053ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:42745 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137222025-04-30T06:43:29.053ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:65114 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137232025-04-30T06:43:29.053ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:38093 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137242025-04-30T06:43:29.053ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:51837 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137252025-04-30T06:43:29.053ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:54850 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137262025-04-30T06:43:29.053ZINFOtest_snapshot_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:54156 remote_addr = [::1]:49092 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137272025-04-30T06:43:29.054ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
137282025-04-30T06:43:29.054ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137292025-04-30T06:43:29.054ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137302025-04-30T06:43:29.054ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
137312025-04-30T06:43:29.054ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137322025-04-30T06:43:29.055ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7f 1c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61389 req_id = 54c265c5-c57f-4c02-93ec-d96932920556
137332025-04-30T06:43:29.055ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32540,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61389 req_id = 54c265c5-c57f-4c02-93ec-d96932920556
137342025-04-30T06:43:29.055ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
137352025-04-30T06:43:29.056ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137362025-04-30T06:43:29.056ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61389 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 54c265c5-c57f-4c02-93ec-d96932920556
137372025-04-30T06:43:29.056ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c549d32-655e-4fa2-97b4-ef850a0dd6b0", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
137382025-04-30T06:43:29.059ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
137392025-04-30T06:43:29.059ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
137402025-04-30T06:43:29.059ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
137412025-04-30T06:43:29.061ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
137422025-04-30T06:43:29.062ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137432025-04-30T06:43:29.062ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137442025-04-30T06:43:29.062ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137452025-04-30T06:43:29.062ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137462025-04-30T06:43:29.063ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137472025-04-30T06:43:29.064ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4273, tv_nsec: 863721727 } })
137482025-04-30T06:43:29.064ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
137492025-04-30T06:43:29.065ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
137502025-04-30T06:43:29.066ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
137512025-04-30T06:43:29.073ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:43372 req_id = 080082b2-e3ab-4205-8034-1d76b5cce321 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137522025-04-30T06:43:29.073ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:42745 req_id = cf2151f4-875e-4a7a-b323-42c89140255f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137532025-04-30T06:43:29.073ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:65114 req_id = 9831895c-5832-46e4-9793-db06c050a8b4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137542025-04-30T06:43:29.073ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:38093 req_id = 906015d6-9fca-4295-9990-bbfc939c7346 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137552025-04-30T06:43:29.073ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:51837 req_id = 633a5280-3fbd-408c-9771-b470cf8ce7bc sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137562025-04-30T06:43:29.073ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:54850 req_id = 5de08582-bf39-41d8-88f8-bd5c24586a77 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137572025-04-30T06:43:29.074ZTRACtest_snapshot_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:54156 method = GET remote_addr = [::1]:49092 req_id = e7557812-b512-4ba8-a509-b6dd3e3fe54c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137582025-04-30T06:43:29.074ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7f 7b 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .{..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55554 req_id = 736b8688-36d3-4986-8aee-feb2ebefda92
137592025-04-30T06:43:29.074ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32635,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55554 req_id = 736b8688-36d3-4986-8aee-feb2ebefda92
137602025-04-30T06:43:29.074ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
137612025-04-30T06:43:29.075ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137622025-04-30T06:43:29.075ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55554 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 736b8688-36d3-4986-8aee-feb2ebefda92
137632025-04-30T06:43:29.075ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 65 90 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 e............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:51104 req_id = 4476ff5b-5d87-4aba-84ad-7d90ede76bca
137642025-04-30T06:43:29.075ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26000,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51104 req_id = 4476ff5b-5d87-4aba-84ad-7d90ede76bca
137652025-04-30T06:43:29.075ZDEBGtest_snapshot_expunge (store): query key key = _dendrite._tcp
137662025-04-30T06:43:29.075ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137672025-04-30T06:43:29.075ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51104 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4476ff5b-5d87-4aba-84ad-7d90ede76bca
137682025-04-30T06:43:29.076ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137692025-04-30T06:43:29.081ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
137702025-04-30T06:43:29.081ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137712025-04-30T06:43:29.081ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137722025-04-30T06:43:29.082ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137732025-04-30T06:43:29.084ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 11768 local_addr = [::1]:54156 method = GET remote_addr = [::1]:43372 req_id = 080082b2-e3ab-4205-8034-1d76b5cce321 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137742025-04-30T06:43:29.084ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 11657 local_addr = [::1]:54156 method = GET remote_addr = [::1]:42745 req_id = cf2151f4-875e-4a7a-b323-42c89140255f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137752025-04-30T06:43:29.084ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 11651 local_addr = [::1]:54156 method = GET remote_addr = [::1]:65114 req_id = 9831895c-5832-46e4-9793-db06c050a8b4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137762025-04-30T06:43:29.085ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 11659 local_addr = [::1]:54156 method = GET remote_addr = [::1]:38093 req_id = 906015d6-9fca-4295-9990-bbfc939c7346 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137772025-04-30T06:43:29.085ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 11559 local_addr = [::1]:54156 method = GET remote_addr = [::1]:51837 req_id = 633a5280-3fbd-408c-9771-b470cf8ce7bc response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137782025-04-30T06:43:29.085ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 11483 local_addr = [::1]:54156 method = GET remote_addr = [::1]:54850 req_id = 5de08582-bf39-41d8-88f8-bd5c24586a77 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137792025-04-30T06:43:29.085ZINFOtest_snapshot_expunge (dropshot): request completed kind = pantry latency_us = 11254 local_addr = [::1]:54156 method = GET remote_addr = [::1]:49092 req_id = e7557812-b512-4ba8-a509-b6dd3e3fe54c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137802025-04-30T06:43:29.089ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4273, tv_nsec: 888547150 } })
137812025-04-30T06:43:29.089ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
137822025-04-30T06:43:29.090ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65174, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4273, tv_nsec: 888960513 } })
137832025-04-30T06:43:29.090ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
137842025-04-30T06:43:29.090ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9f 53 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .S..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50107 req_id = 4e36aa5a-c02d-4094-a86d-e44880afa8d6
137852025-04-30T06:43:29.090ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40787,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50107 req_id = 4e36aa5a-c02d-4094-a86d-e44880afa8d6
137862025-04-30T06:43:29.090ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137872025-04-30T06:43:29.090ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50107 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4e36aa5a-c02d-4094-a86d-e44880afa8d6
137882025-04-30T06:43:29.092ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137892025-04-30T06:43:29.097ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137902025-04-30T06:43:29.097ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137912025-04-30T06:43:29.097ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
137922025-04-30T06:43:29.104ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
137932025-04-30T06:43:29.176ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
137942025-04-30T06:43:29.176ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
137952025-04-30T06:43:29.176ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
137962025-04-30T06:43:29.176ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
137972025-04-30T06:43:29.177ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
137982025-04-30T06:43:29.178ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
137992025-04-30T06:43:29.178ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
138002025-04-30T06:43:29.189ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 91 dc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55904 req_id = 5f580c98-4a54-48d1-8daf-646839d11560
138012025-04-30T06:43:29.189ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37340,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55904 req_id = 5f580c98-4a54-48d1-8daf-646839d11560
138022025-04-30T06:43:29.189ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138032025-04-30T06:43:29.189ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55904 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5f580c98-4a54-48d1-8daf-646839d11560
138042025-04-30T06:43:29.189ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 82 e0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50712 req_id = bfc84440-3a35-4527-bb74-af10a26d91ef
138052025-04-30T06:43:29.189ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33504,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50712 req_id = bfc84440-3a35-4527-bb74-af10a26d91ef
138062025-04-30T06:43:29.189ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138072025-04-30T06:43:29.190ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50712 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bfc84440-3a35-4527-bb74-af10a26d91ef
138082025-04-30T06:43:29.190ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
138092025-04-30T06:43:29.191ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
138102025-04-30T06:43:29.191ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
138112025-04-30T06:43:29.191ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
138122025-04-30T06:43:29.270ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
138132025-04-30T06:43:29.271ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
138142025-04-30T06:43:29.271ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
138152025-04-30T06:43:29.271ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
138162025-04-30T06:43:29.276ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
138172025-04-30T06:43:29.276ZINFOtest_snapshot_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
138182025-04-30T06:43:29.357ZINFOtest_snapshot_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
138192025-04-30T06:43:29.358ZDEBGtest_snapshot_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
138202025-04-30T06:43:29.359ZDEBGtest_snapshot_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
138212025-04-30T06:43:29.359ZWARNtest_snapshot_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
138222025-04-30T06:43:29.363ZWARNtest_snapshot_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
138232025-04-30T06:43:29.416ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
138242025-04-30T06:43:29.417ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
138252025-04-30T06:43:29.420ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.484077647s iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
138262025-04-30T06:43:29.420ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
138272025-04-30T06:43:29.420ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
138282025-04-30T06:43:29.420ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138292025-04-30T06:43:29.420ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138302025-04-30T06:43:29.420ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138312025-04-30T06:43:29.421ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138322025-04-30T06:43:29.423ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
138332025-04-30T06:43:29.426ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138342025-04-30T06:43:29.432ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
138352025-04-30T06:43:29.432ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138362025-04-30T06:43:29.432ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138372025-04-30T06:43:29.433ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138382025-04-30T06:43:29.455ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
138392025-04-30T06:43:29.456ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
138402025-04-30T06:43:29.463ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fa 7c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .|..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55800 req_id = 4a1102df-3ee6-43db-803c-0aed3d06f65b
138412025-04-30T06:43:29.464ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64124,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55800 req_id = 4a1102df-3ee6-43db-803c-0aed3d06f65b
138422025-04-30T06:43:29.465ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
138432025-04-30T06:43:29.466ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138442025-04-30T06:43:29.466ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55800 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4a1102df-3ee6-43db-803c-0aed3d06f65b
138452025-04-30T06:43:29.470ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4274, tv_nsec: 268098722 } })
138462025-04-30T06:43:29.470ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
138472025-04-30T06:43:29.470ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: db 7a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .z..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49832 req_id = 54296e8a-75a7-4aad-bede-ee7f9a85aaef
138482025-04-30T06:43:29.470ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56186,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49832 req_id = 54296e8a-75a7-4aad-bede-ee7f9a85aaef
138492025-04-30T06:43:29.470ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138502025-04-30T06:43:29.470ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49832 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 54296e8a-75a7-4aad-bede-ee7f9a85aaef
138512025-04-30T06:43:29.530ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
138522025-04-30T06:43:29.532ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138532025-04-30T06:43:29.532ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138542025-04-30T06:43:29.532ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138552025-04-30T06:43:29.532ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
138562025-04-30T06:43:29.533ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138572025-04-30T06:43:29.533ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 28 9b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c (............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62535 req_id = 4e515804-26b1-4a06-8a8d-d6cc72be6276
138582025-04-30T06:43:29.534ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10395,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62535 req_id = 4e515804-26b1-4a06-8a8d-d6cc72be6276
138592025-04-30T06:43:29.534ZDEBGtest_snapshot_expunge (store): query key key = _clickhouse-admin-keeper._tcp
138602025-04-30T06:43:29.534ZERROtest_snapshot_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10395, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62535 req_id = 4e515804-26b1-4a06-8a8d-d6cc72be6276
138612025-04-30T06:43:29.536ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection background_task = inventory_collection
138622025-04-30T06:43:29.536ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40004
138632025-04-30T06:43:29.536ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/ignition
138642025-04-30T06:43:29.536ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138652025-04-30T06:43:29.548ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138662025-04-30T06:43:29.549ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138672025-04-30T06:43:29.549ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
138682025-04-30T06:43:29.549ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138692025-04-30T06:43:29.553ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138702025-04-30T06:43:29.554ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
138712025-04-30T06:43:29.556ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138722025-04-30T06:43:29.564ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddae7955-ae20-4b84-877e-021950f38f43", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
138732025-04-30T06:43:29.565ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0
138742025-04-30T06:43:29.574ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d7b1f9d-64f4-4255-9564-476d504d7656", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
138752025-04-30T06:43:29.575ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0
138762025-04-30T06:43:29.583ZINFOtest_snapshot_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
138772025-04-30T06:43:29.583ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138782025-04-30T06:43:29.584ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75d9687a-0c14-402b-b895-8563632d5e50 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138792025-04-30T06:43:29.585ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75d9687a-0c14-402b-b895-8563632d5e50 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138802025-04-30T06:43:29.585ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75d9687a-0c14-402b-b895-8563632d5e50 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138812025-04-30T06:43:29.589ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75d9687a-0c14-402b-b895-8563632d5e50 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138822025-04-30T06:43:29.593ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 8537 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = 75d9687a-0c14-402b-b895-8563632d5e50 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138832025-04-30T06:43:29.594ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "75d9687a-0c14-402b-b895-8563632d5e50", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
138842025-04-30T06:43:29.595ZDEBGtest_snapshot_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138852025-04-30T06:43:29.597ZTRACtest_snapshot_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = abba5503-b521-45c6-8fdf-ede7347d5097 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138862025-04-30T06:43:29.597ZTRACtest_snapshot_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = abba5503-b521-45c6-8fdf-ede7347d5097 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138872025-04-30T06:43:29.597ZDEBGtest_snapshot_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = abba5503-b521-45c6-8fdf-ede7347d5097 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138882025-04-30T06:43:29.598ZDEBGtest_snapshot_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = abba5503-b521-45c6-8fdf-ede7347d5097 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138892025-04-30T06:43:29.601ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b361b14b-5eb5-4975-8a4d-be764cc1f4d2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
138902025-04-30T06:43:29.602ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1
138912025-04-30T06:43:29.611ZINFOtest_snapshot_expunge (dropshot_internal): request completed latency_us = 14701 local_addr = 127.0.0.1:41836 method = GET remote_addr = 127.0.0.1:57068 req_id = abba5503-b521-45c6-8fdf-ede7347d5097 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138922025-04-30T06:43:29.629ZDEBGtest_snapshot_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:41836/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "abba5503-b521-45c6-8fdf-ede7347d5097", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
138932025-04-30T06:43:29.629ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 58f2365f-b720-4ec1-ad50-a68fadcacb1c
138942025-04-30T06:43:29.629ZTRACtest_snapshot_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138952025-04-30T06:43:29.629ZINFOtest_snapshot_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
138962025-04-30T06:43:29.629ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
138972025-04-30T06:43:29.633ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2459670c-11d7-4ed2-bca8-c533479dacd0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
138982025-04-30T06:43:29.634ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0
138992025-04-30T06:43:29.638ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d717369-4773-4a4d-8728-9200f13b665d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139002025-04-30T06:43:29.638ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1
139012025-04-30T06:43:29.641ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2846d89a-378d-4cbc-88af-59a8ef6bc250", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139022025-04-30T06:43:29.641ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0
139032025-04-30T06:43:29.647ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96ec964f-d1cc-4b2b-afaf-8d65558060ef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139042025-04-30T06:43:29.648ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1
139052025-04-30T06:43:29.650ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27ee0e4a-cdb4-4aee-a39a-31bdcc431388", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139062025-04-30T06:43:29.650ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cmpa
139072025-04-30T06:43:29.651ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b484daf4-33a7-40d4-b4d4-5a17d8467c90", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139082025-04-30T06:43:29.651ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
139092025-04-30T06:43:29.652ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54e53f14-7ad1-40a4-8f56-16b35178f2e6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139102025-04-30T06:43:29.653ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
139112025-04-30T06:43:29.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9d10d05-f2e9-4c23-8293-fd5323997110", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139122025-04-30T06:43:29.655ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/0/component/rot/cfpa
139132025-04-30T06:43:29.657ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f0400cd-2229-47f0-81f5-736697dbf541", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139142025-04-30T06:43:29.658ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1
139152025-04-30T06:43:29.660ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31c05bd5-115d-4bae-b557-8d7143711888", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139162025-04-30T06:43:29.661ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0
139172025-04-30T06:43:29.665ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a39bfe78-7d0d-461e-8abe-c30888c03920", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139182025-04-30T06:43:29.665ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1
139192025-04-30T06:43:29.669ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0edfb6df-9186-4699-8939-132bacca5f99", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139202025-04-30T06:43:29.669ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0
139212025-04-30T06:43:29.673ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b11548b-49d3-4d74-9bcb-ea69bed2cdc7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139222025-04-30T06:43:29.673ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1
139232025-04-30T06:43:29.676ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "149d610c-0355-44a5-b4a6-7f4332dcc2b3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139242025-04-30T06:43:29.676ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0
139252025-04-30T06:43:29.681ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
139262025-04-30T06:43:29.681ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139272025-04-30T06:43:29.681ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139282025-04-30T06:43:29.681ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
139292025-04-30T06:43:29.682ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139302025-04-30T06:43:29.684ZDEBGtest_snapshot_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
139312025-04-30T06:43:29.684ZDEBGtest_snapshot_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
139322025-04-30T06:43:29.684ZDEBGtest_snapshot_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6
139332025-04-30T06:43:29.686ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
139342025-04-30T06:43:29.686ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de2b846d-5620-435c-abb4-09fe0d0db82d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139352025-04-30T06:43:29.686ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1
139362025-04-30T06:43:29.688ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58f566d5-0103-41b6-94cb-009df3d36a32", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139372025-04-30T06:43:29.688ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cmpa
139382025-04-30T06:43:29.689ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139392025-04-30T06:43:29.700ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
139402025-04-30T06:43:29.700ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139412025-04-30T06:43:29.700ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
139422025-04-30T06:43:29.701ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139432025-04-30T06:43:29.701ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 2 table_name = fields_string
139442025-04-30T06:43:29.701ZTRACtest_snapshot_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
139452025-04-30T06:43:29.704ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64c70679-caa3-4821-a9c6-1712d81cf51b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139462025-04-30T06:43:29.704ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
139472025-04-30T06:43:29.705ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139482025-04-30T06:43:29.706ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
139492025-04-30T06:43:29.708ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139502025-04-30T06:43:29.708ZDEBGtest_snapshot_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f2d4d5c2-3547-4831-8529-796764280fb6 n_rows = 1 table_name = measurements_i64
139512025-04-30T06:43:29.708ZTRACtest_snapshot_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
139522025-04-30T06:43:29.711ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3aca76e-9fa0-43f2-905b-c3baaf10a5f9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139532025-04-30T06:43:29.713ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
139542025-04-30T06:43:29.716ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e09c1707-c923-46a6-b968-1cd37544e84b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139552025-04-30T06:43:29.716ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/switch/1/component/rot/cfpa
139562025-04-30T06:43:29.719ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dc1f4c3-b179-441e-8176-0ea6540d0d9a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139572025-04-30T06:43:29.719ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0
139582025-04-30T06:43:29.722ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbb05f77-2233-4fcd-a14d-a66b68642734", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139592025-04-30T06:43:29.722ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0
139602025-04-30T06:43:29.723ZDEBGtest_snapshot_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139612025-04-30T06:43:29.723ZDEBGtest_snapshot_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139622025-04-30T06:43:29.727ZTRACtest_snapshot_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139632025-04-30T06:43:29.727ZDEBGtest_snapshot_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139642025-04-30T06:43:29.727ZTRACtest_snapshot_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139652025-04-30T06:43:29.727ZTRACtest_snapshot_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139662025-04-30T06:43:29.730ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63746ccc-4a6c-42b7-9d27-1ef6c3044a75", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139672025-04-30T06:43:29.730ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1
139682025-04-30T06:43:29.734ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50e8ee44-f6eb-4e0f-8b6f-637d074be445", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139692025-04-30T06:43:29.734ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0
139702025-04-30T06:43:29.737ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9986f26f-fabf-436e-af4c-e29177951ed6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139712025-04-30T06:43:29.737ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1
139722025-04-30T06:43:29.742ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38b4894c-4023-40cb-850e-eb3dd48d3631", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139732025-04-30T06:43:29.742ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0
139742025-04-30T06:43:29.744ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cabbb8a4-ee72-4b17-8c8c-94b4a0f47182", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139752025-04-30T06:43:29.744ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1
139762025-04-30T06:43:29.747ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e6643aa-9d5e-4a2f-affd-f199ab933bc9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139772025-04-30T06:43:29.747ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cmpa
139782025-04-30T06:43:29.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "775c9bb6-be4c-4499-bf51-e277820b7412", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139792025-04-30T06:43:29.750ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
139802025-04-30T06:43:29.753ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
139812025-04-30T06:43:29.753ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7107e36-35eb-4b1d-8f40-303b87c7d39b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139822025-04-30T06:43:29.753ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
139832025-04-30T06:43:29.754ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c037546c-b041-4c8d-bb15-afe7601a3122", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139842025-04-30T06:43:29.754ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/0/component/rot/cfpa
139852025-04-30T06:43:29.756ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcd3a14b-0211-4d8f-808b-3a22b9351fc3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139862025-04-30T06:43:29.756ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1
139872025-04-30T06:43:29.758ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ac608f4-ab3e-4970-bf9d-d7f177ef305d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139882025-04-30T06:43:29.759ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0
139892025-04-30T06:43:29.763ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0390686f-b519-4481-bf70-3e0b3e0dc847", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139902025-04-30T06:43:29.764ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1
139912025-04-30T06:43:29.767ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f655f70-ad72-43c0-80d3-1446e7953479", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139922025-04-30T06:43:29.767ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0
139932025-04-30T06:43:29.770ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f535b671-f7f4-4cc1-a7fd-09f9b518fb45", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139942025-04-30T06:43:29.770ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1
139952025-04-30T06:43:29.772ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c15239f-627c-4ecb-aeca-35ae7736305c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139962025-04-30T06:43:29.772ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0
139972025-04-30T06:43:29.777ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b0ce818-3ca8-4506-b0b0-86f735314cad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
139982025-04-30T06:43:29.777ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1
139992025-04-30T06:43:29.782ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "351d2148-39d7-4e89-9e7a-62f39c35b9d8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
140002025-04-30T06:43:29.782ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cmpa
140012025-04-30T06:43:29.784ZWARNtest_snapshot_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
140022025-04-30T06:43:29.785ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10e56f6e-7031-4a94-a749-916744f68f03", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
140032025-04-30T06:43:29.785ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
140042025-04-30T06:43:29.786ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd370214-976b-45ac-83b9-72a6e0e9377e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
140052025-04-30T06:43:29.786ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
140062025-04-30T06:43:29.788ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bdb8f14-3aaf-4e36-8676-89f2885a60dc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
140072025-04-30T06:43:29.788ZDEBGtest_snapshot_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40004 method = GET uri = http://[::1]:40004/sp/sled/1/component/rot/cfpa
140082025-04-30T06:43:29.789ZDEBGtest_snapshot_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40004 result = Ok(Response { url: "http://[::1]:40004/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bee500d-0181-4c51-9698-b2982513d29b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
140092025-04-30T06:43:29.790ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140102025-04-30T06:43:29.790ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140112025-04-30T06:43:29.790ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140122025-04-30T06:43:29.790ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140132025-04-30T06:43:29.790ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140142025-04-30T06:43:29.793ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140152025-04-30T06:43:29.800ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140162025-04-30T06:43:29.800ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140172025-04-30T06:43:29.800ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140182025-04-30T06:43:29.800ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140192025-04-30T06:43:29.800ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140202025-04-30T06:43:29.808ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140212025-04-30T06:43:29.808ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
140222025-04-30T06:43:29.808ZTRACtest_snapshot_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140232025-04-30T06:43:29.808ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140242025-04-30T06:43:29.808ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
140252025-04-30T06:43:29.809ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140262025-04-30T06:43:29.814ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140272025-04-30T06:43:29.822ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140282025-04-30T06:43:29.823ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140292025-04-30T06:43:29.823ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
140302025-04-30T06:43:29.824ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140312025-04-30T06:43:29.825ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140322025-04-30T06:43:29.831ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140332025-04-30T06:43:29.832ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140342025-04-30T06:43:29.832ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140352025-04-30T06:43:29.834ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140362025-04-30T06:43:29.840ZTRACtest_snapshot_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140372025-04-30T06:43:29.841ZDEBGtest_snapshot_expunge (InternalApi): roles roles = RoleSet { roles: {} }
140382025-04-30T06:43:29.843ZDEBGtest_snapshot_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140392025-04-30T06:43:29.937ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:51119
140402025-04-30T06:43:29.937ZDEBGtest_snapshot_expunge (BackgroundTasks): client request SledAgent = http://[::1]:51119 background_task = inventory_collection body = None method = GET uri = http://[::1]:51119/inventory
140412025-04-30T06:43:29.938ZINFOtest_snapshot_expunge (dropshot): accepted connection local_addr = [::1]:51119 remote_addr = [::1]:53051 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140422025-04-30T06:43:29.939ZTRACtest_snapshot_expunge (dropshot): incoming request local_addr = [::1]:51119 method = GET remote_addr = [::1]:53051 req_id = 91de45a9-6568-4036-addb-806ec9fba24a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
140432025-04-30T06:43:29.939ZINFOtest_snapshot_expunge (dropshot): request completed latency_us = 501 local_addr = [::1]:51119 method = GET remote_addr = [::1]:53051 req_id = 91de45a9-6568-4036-addb-806ec9fba24a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
140442025-04-30T06:43:29.940ZDEBGtest_snapshot_expunge (BackgroundTasks): client response SledAgent = http://[::1]:51119 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:51119/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "91de45a9-6568-4036-addb-806ec9fba24a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
140452025-04-30T06:43:29.943ZDEBGtest_snapshot_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
140462025-04-30T06:43:29.943ZDEBGtest_snapshot_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
140472025-04-30T06:43:29.943ZDEBGtest_snapshot_expunge (BackgroundTasks): finished collection background_task = inventory_collection
140482025-04-30T06:43:29.946ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
140492025-04-30T06:43:29.946ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140502025-04-30T06:43:29.946ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140512025-04-30T06:43:29.946ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140522025-04-30T06:43:29.947ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140532025-04-30T06:43:29.952ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140542025-04-30T06:43:29.957ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
140552025-04-30T06:43:29.957ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140562025-04-30T06:43:29.958ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140572025-04-30T06:43:29.958ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140582025-04-30T06:43:30.014ZDEBGtest_snapshot_expunge: attempt to fetch latest inventory collection result = Ok("found")
140592025-04-30T06:43:30.035ZWARNtest_snapshot_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
140602025-04-30T06:43:30.035ZWARNtest_snapshot_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
140612025-04-30T06:43:30.036ZDEBGtest_snapshot_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 615.228694ms iteration = 50 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
140622025-04-30T06:43:30.036ZDEBGtest_snapshot_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 51 reason = Signaled
140632025-04-30T06:43:30.036ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
140642025-04-30T06:43:30.036ZTRACtest_snapshot_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140652025-04-30T06:43:30.036ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140662025-04-30T06:43:30.036ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140672025-04-30T06:43:30.038ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140682025-04-30T06:43:30.045ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140692025-04-30T06:43:30.051ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
140702025-04-30T06:43:30.051ZTRACtest_snapshot_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140712025-04-30T06:43:30.051ZDEBGtest_snapshot_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140722025-04-30T06:43:30.052ZDEBGtest_snapshot_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140732025-04-30T06:43:30.060ZDEBGtest_snapshot_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(b9eba9a5-2952-4b7f-9d32-4296e0270f0a, 0)]
140742025-04-30T06:43:30.060ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
140752025-04-30T06:43:30.061ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5f bf 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 _............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54435 req_id = 5d286383-a5b4-4896-aefa-e1bee46a1583
140762025-04-30T06:43:30.061ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24511,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54435 req_id = 5d286383-a5b4-4896-aefa-e1bee46a1583
140772025-04-30T06:43:30.062ZDEBGtest_snapshot_expunge (store): query key key = _mgs._tcp
140782025-04-30T06:43:30.062ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
140792025-04-30T06:43:30.062ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54435 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5d286383-a5b4-4896-aefa-e1bee46a1583
140802025-04-30T06:43:30.063ZDEBGtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40004, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4274, tv_nsec: 862490348 } })
140812025-04-30T06:43:30.063ZTRACtest_snapshot_expunge (DnsResolver): lookup_service_targets: looking up SRV target
140822025-04-30T06:43:30.064ZTRACtest_snapshot_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9f 84 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58315 req_id = ef8f209e-9810-4580-b7b6-26be3b1ee24a
140832025-04-30T06:43:30.065ZDEBGtest_snapshot_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40836,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58315 req_id = ef8f209e-9810-4580-b7b6-26be3b1ee24a
140842025-04-30T06:43:30.065ZDEBGtest_snapshot_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
140852025-04-30T06:43:30.065ZDEBGtest_snapshot_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58315 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ef8f209e-9810-4580-b7b6-26be3b1ee24a
140862025-04-30T06:43:30.147ZTRACtest_snapshot_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal