Max level shown:
12025-04-30T06:42:12.719ZDEBGtest_delete_snapshot (init_with_steps): Running step start_crdb
22025-04-30T06:42:12.720ZDEBGtest_delete_snapshot: Starting CRDB
32025-04-30T06:42:12.758ZINFOtest_delete_snapshot: cockroach temporary directory: /var/tmp/omicron_tmp/.tmpe57Qkr
42025-04-30T06:42:12.758ZINFOtest_delete_snapshot: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmpe57Qkr/data)
52025-04-30T06:42:12.792ZINFOtest_delete_snapshot: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmpe57Qkr/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmpe57Qkr/listen-url
62025-04-30T06:42:12.793ZINFOtest_delete_snapshot: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=6 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:42:14.271ZINFOtest_delete_snapshot: cockroach pid: 26868
82025-04-30T06:42:14.271ZINFOtest_delete_snapshot: cockroach listen URL: postgresql://root@[::1]:34703/omicron?sslmode=disable
92025-04-30T06:42:14.271ZDEBGtest_delete_snapshot (init_with_steps): Running step start_clickhouse
102025-04-30T06:42:14.271ZDEBGtest_delete_snapshot: Starting Clickhouse
112025-04-30T06:42:14.971ZDEBGtest_delete_snapshot (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:42:14.971ZDEBGtest_delete_snapshot: Starting Management Gateway
132025-04-30T06:42:15.094ZDEBGtest_delete_snapshot (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:42:15.094ZDEBGtest_delete_snapshot: Starting Dendrite for switch0
152025-04-30T06:42:15.141ZDEBGtest_delete_snapshot (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:42:15.141ZDEBGtest_delete_snapshot: Starting mgd for switch0
172025-04-30T06:42:15.257ZDEBGtest_delete_snapshot: mgd port is 50898
182025-04-30T06:42:15.257ZDEBGtest_delete_snapshot (init_with_steps): Running step record_switch_dns
192025-04-30T06:42:15.257ZDEBGtest_delete_snapshot: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:42:15.258ZDEBGtest_delete_snapshot (init_with_steps): Running step start_internal_dns
212025-04-30T06:42:15.264ZINFOtest_delete_snapshot (store): opening sled database path = /var/tmp/omicron_tmp/.tmpPU7D21
222025-04-30T06:42:15.435ZINFOtest_delete_snapshot (store): pruning trees for generations newer than 0
232025-04-30T06:42:15.435ZINFOtest_delete_snapshot (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:42:15.435ZINFOtest_delete_snapshot (dns): DNS server bound to address kind = dns local_address = [::1]:47519
252025-04-30T06:42:15.451ZDEBGtest_delete_snapshot (http): registered endpoint kind = dns local_addr = [::1]:51007 method = GET path = /config versions = all
262025-04-30T06:42:15.451ZDEBGtest_delete_snapshot (http): registered endpoint kind = dns local_addr = [::1]:51007 method = PUT path = /config versions = all
272025-04-30T06:42:15.451ZINFOtest_delete_snapshot (http): listening kind = dns local_addr = [::1]:51007
282025-04-30T06:42:15.452ZDEBGtest_delete_snapshot (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:51007
292025-04-30T06:42:15.452ZDEBGtest_delete_snapshot (init_with_steps): Running step start_external_dns
302025-04-30T06:42:15.452ZINFOtest_delete_snapshot (store): opening sled database path = /var/tmp/omicron_tmp/.tmpdvvHBp
312025-04-30T06:42:15.469ZINFOtest_delete_snapshot (store): pruning trees for generations newer than 0
322025-04-30T06:42:15.469ZINFOtest_delete_snapshot (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:42:15.469ZINFOtest_delete_snapshot (dns): DNS server bound to address kind = dns local_address = [::1]:53686
342025-04-30T06:42:15.469ZDEBGtest_delete_snapshot (http): registered endpoint kind = dns local_addr = [::1]:41697 method = GET path = /config versions = all
352025-04-30T06:42:15.469ZDEBGtest_delete_snapshot (http): registered endpoint kind = dns local_addr = [::1]:41697 method = PUT path = /config versions = all
362025-04-30T06:42:15.469ZINFOtest_delete_snapshot (http): listening kind = dns local_addr = [::1]:41697
372025-04-30T06:42:15.472ZDEBGtest_delete_snapshot (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:41697
382025-04-30T06:42:15.472ZDEBGtest_delete_snapshot (init_with_steps): Running step start_nexus_internal
392025-04-30T06:42:15.472ZDEBGtest_delete_snapshot: Starting Nexus (internal API)
402025-04-30T06:42:15.480ZINFOtest_delete_snapshot: setting up nexus server
412025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Action
422025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Database
462025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Fleet
482025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Inventory
492025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:42:15.510ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:42:15.511ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Project
582025-04-30T06:42:15.511ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Disk
592025-04-30T06:42:15.511ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:42:15.511ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:42:15.511ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:42:15.511ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:42:15.511ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Instance
642025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = IpPool
652025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Vpc
672025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Image
752025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Certificate
802025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:42:15.512ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:42:15.519ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:42:15.519ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:42:15.519ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Rack
852025-04-30T06:42:15.519ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:42:15.520ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = SshKey
872025-04-30T06:42:15.520ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Silo
882025-04-30T06:42:15.520ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:42:15.520ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:42:15.521ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:42:15.521ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:42:15.521ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:42:15.521ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Sled
942025-04-30T06:42:15.522ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:42:15.522ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:42:15.522ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:42:15.522ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:42:15.522ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:42:15.522ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:42:15.524ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = Service
1012025-04-30T06:42:15.524ZINFOtest_delete_snapshot (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:42:15.524ZINFOtest_delete_snapshot (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:42:15.566ZINFOtest_delete_snapshot (ServerContext): Setting up resolver using DNS address: [::1]:47519
1042025-04-30T06:42:15.566ZINFOtest_delete_snapshot (DnsResolver): new DNS resolver addresses = [[::1]:47519]
1052025-04-30T06:42:15.567ZINFOtest_delete_snapshot (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:34703/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 34703,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:42:15.569ZDEBGtest_delete_snapshot (ServerContext): registered USDT probes
1072025-04-30T06:42:15.653ZINFOtest_delete_snapshot (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:42:15.891ZDEBGtest_delete_snapshot (clickhouse-client): registered USDT probes id = dbb2cc0d-b0d3-4188-8fe9-9a9e838660f4
1092025-04-30T06:42:15.971ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /bgtasks versions = all
1102025-04-30T06:42:15.981ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:42:15.981ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:42:15.981ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /demo-saga versions = all
1202025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:42:15.982ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /metrics/producers versions = all
1442025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /mgs-updates versions = all
1452025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /sagas versions = all
1522025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /sleds/add versions = all
1572025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = GET path = /v1/ping versions = all
1612025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): registered endpoint local_addr = 127.0.0.1:35404 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:42:15.983ZINFOtest_delete_snapshot (dropshot_internal): listening local_addr = 127.0.0.1:35404
1642025-04-30T06:42:15.983ZDEBGtest_delete_snapshot (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:35404
1652025-04-30T06:42:15.984ZINFOtest_delete_snapshot (SEC): SEC running sec_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
1662025-04-30T06:42:15.984ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:42:15.984ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:42:15.988ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:42:15.989ZDEBGtest_delete_snapshot (DataLoader): attempting to create built-in users
1702025-04-30T06:42:15.990ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:42:15.990ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:42:15.990ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:42:15.991ZINFOtest_delete_snapshot (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:42:15.997ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: d0 f4 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:60483 req_id = 52626544-6e00-43c5-8784-456fab361bbc
1752025-04-30T06:42:15.997ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53492,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60483 req_id = 52626544-6e00-43c5-8784-456fab361bbc
1762025-04-30T06:42:16.005ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53492, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60483 req_id = 52626544-6e00-43c5-8784-456fab361bbc
1772025-04-30T06:42:16.030ZINFOtest_delete_snapshot (DataLoader): created 5 built-in users
1782025-04-30T06:42:16.031ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:42:16.031ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:42:16.034ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:42:16.034ZDEBGtest_delete_snapshot (DataLoader): attempting to create built-in roles
1822025-04-30T06:42:16.034ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:42:16.034ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:42:16.035ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:42:16.132ZINFOtest_delete_snapshot (DataLoader): created 10 built-in roles
1862025-04-30T06:42:16.132ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:42:16.132ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:42:16.135ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:42:16.135ZDEBGtest_delete_snapshot (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:42:16.135ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:42:16.135ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:42:16.135ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:42:16.477ZINFOtest_delete_snapshot (DataLoader): created 4 built-in role assignments
1942025-04-30T06:42:16.477ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:42:16.477ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:42:16.479ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:42:16.479ZDEBGtest_delete_snapshot (DataLoader): attempting to create built-in silos
1982025-04-30T06:42:16.480ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:42:16.480ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:42:16.480ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:42:16.572ZINFOtest_delete_snapshot (DataLoader): created 2 built-in silos
2022025-04-30T06:42:16.572ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:42:16.572ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:42:16.573ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:42:16.584ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:42:16.584ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:42:16.586ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:42:16.586ZDEBGtest_delete_snapshot (DataLoader): attempting to create built-in projects
2092025-04-30T06:42:16.586ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:42:16.586ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:42:16.586ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:42:16.591ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:42:16.591ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:42:16.591ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:42:16.591ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:42:16.592ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:42:16.646ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:42:16.646ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:42:16.646ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:42:16.646ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:42:16.649ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:42:16.652ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:42:16.652ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:42:16.652ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:42:16.652ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:42:16.652ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:42:16.653ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:42:16.655ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:42:16.655ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:42:16.655ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:42:16.656ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:42:16.659ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:42:16.662ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:42:16.662ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:42:16.662ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:42:16.662ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:42:16.685ZINFOtest_delete_snapshot (DataLoader): created built-in services project
2382025-04-30T06:42:16.685ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:42:16.685ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:42:16.688ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:42:16.688ZDEBGtest_delete_snapshot (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:42:16.688ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:42:16.688ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:42:16.689ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:42:16.694ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:42:16.694ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:42:16.695ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:42:16.697ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:42:16.697ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:42:16.697ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:42:16.697ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:42:16.698ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:42:16.701ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:42:16.701ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:42:16.701ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:42:16.701ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:42:16.702ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:42:16.702ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:42:16.702ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:42:16.702ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:42:16.705ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:42:16.711ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:42:16.711ZTRACtest_delete_snapshot (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:42:16.711ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:42:16.711ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:42:16.711ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:42:16.711ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:42:16.712ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:42:16.714ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:42:16.714ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:42:16.714ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:42:16.714ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:42:16.716ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:42:16.716ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:42:16.716ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:42:16.717ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:42:16.719ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:42:16.724ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:42:16.724ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:42:16.724ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:42:16.725ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:42:16.738ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:42:16.738ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:42:16.738ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:42:16.738ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:42:16.738ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:42:16.741ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:42:16.741ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:42:16.741ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:42:16.741ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:42:16.744ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:42:16.745ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:42:16.745ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:42:16.745ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:42:16.747ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:42:16.752ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:42:16.752ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:42:16.752ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:42:16.753ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:42:16.760ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:42:16.760ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:42:16.761ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:42:16.768ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:42:16.768ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:42:16.769ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:42:16.774ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:42:16.775ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:42:16.775ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:42:16.775ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:42:16.775ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:42:16.788ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:42:16.789ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:42:16.789ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:42:16.789ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:42:16.792ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:42:16.792ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:42:16.792ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:42:16.792ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:42:16.794ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:42:16.799ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:42:16.799ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:42:16.799ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:42:16.800ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:42:16.895ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:42:16.895ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:42:16.895ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:42:16.895ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:42:16.896ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:42:16.900ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:42:16.901ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:42:16.901ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:42:16.901ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:42:16.903ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:42:16.903ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:42:16.903ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:42:16.904ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:42:16.907ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:42:16.913ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:42:16.913ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:42:16.913ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:42:16.914ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:42:16.969ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:42:16.969ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:42:16.969ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:42:16.970ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:42:16.971ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:42:16.975ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:42:16.975ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:42:16.975ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:42:16.976ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:42:16.981ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:42:16.981ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:42:16.981ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:42:16.983ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:42:16.993ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:42:17.011ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:42:17.011ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:42:17.011ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:42:17.014ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:42:17.033ZDEBGtest_delete_snapshot (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:42:17.033ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:42:17.034ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:42:17.036ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:42:17.041ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:42:17.041ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:42:17.041ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:42:17.045ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:42:17.045ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:42:17.045ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:42:17.048ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:42:17.048ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:42:17.048ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:42:17.048ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:42:17.049ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:42:17.052ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:42:17.052ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:42:17.052ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:42:17.052ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:42:17.055ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:42:17.055ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:42:17.055ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:42:17.056ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:42:17.058ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:42:17.064ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:42:17.064ZTRACtest_delete_snapshot (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:42:17.064ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:42:17.064ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:42:17.064ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:42:17.064ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:42:17.072ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:42:17.074ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:42:17.074ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:42:17.074ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:42:17.076ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:42:17.082ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:42:17.082ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:42:17.082ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:42:17.082ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:42:17.095ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:42:17.112ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:42:17.112ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:42:17.112ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:42:17.113ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:42:17.128ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:42:17.128ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:42:17.128ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:42:17.128ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:42:17.129ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:42:17.137ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:42:17.137ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:42:17.137ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:42:17.137ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:42:17.148ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:42:17.148ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:42:17.148ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:42:17.149ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:42:17.153ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:42:17.161ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:42:17.161ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:42:17.161ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:42:17.162ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:42:17.244ZINFOtest_delete_snapshot (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:42:17.244ZDEBGtest_delete_snapshot (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:42:17.244ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:42:17.244ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:42:17.245ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:42:17.251ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:42:17.251ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:42:17.253ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:42:17.255ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:42:17.256ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:42:17.256ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:42:17.261ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:42:17.261ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:42:17.261ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:42:17.261ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:42:17.262ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:42:17.265ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:42:17.265ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:42:17.265ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:42:17.266ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:42:17.269ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:42:17.269ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:42:17.269ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:42:17.270ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:42:17.273ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:42:17.279ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:42:17.279ZTRACtest_delete_snapshot (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:42:17.279ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:42:17.279ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:42:17.280ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:42:17.299ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:42:17.299ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:42:17.299ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:42:17.299ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:42:17.300ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:42:17.310ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:42:17.311ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:42:17.311ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:42:17.311ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:42:17.376ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:42:17.376ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:42:17.376ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:42:17.377ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:42:17.379ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:42:17.384ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:42:17.391ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:42:17.391ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:42:17.391ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:42:17.391ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:42:17.392ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:42:17.394ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:42:17.394ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:42:17.395ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:42:17.395ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:42:17.397ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:42:17.397ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:42:17.397ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:42:17.397ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:42:17.399ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:42:17.406ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:42:17.406ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:42:17.406ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:42:17.406ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:42:17.472ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:42:17.473ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:42:17.474ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:42:17.479ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:42:17.479ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:42:17.479ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:42:17.479ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:42:17.480ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:42:17.482ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:42:17.482ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:42:17.482ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:42:17.482ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:42:17.486ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:42:17.486ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:42:17.486ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:42:17.486ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:42:17.488ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:42:17.495ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:42:17.500ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:42:17.500ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:42:17.500ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:42:17.500ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:42:17.501ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:42:17.503ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:42:17.503ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:42:17.503ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:42:17.504ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:42:17.505ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:42:17.505ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:42:17.505ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:42:17.506ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:42:17.508ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:42:17.514ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:42:17.514ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:42:17.514ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:42:17.515ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:42:17.521ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:42:17.521ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:42:17.521ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:42:17.523ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:42:17.523ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:42:17.524ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:42:17.524ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:42:17.524ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:42:17.527ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:42:17.527ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:42:17.527ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:42:17.528ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:42:17.530ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:42:17.530ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:42:17.530ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:42:17.531ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:42:17.533ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:42:17.552ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:42:17.562ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:42:17.562ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:42:17.562ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:42:17.562ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:42:17.563ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:42:17.566ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:42:17.566ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:42:17.566ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:42:17.567ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:42:17.569ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:42:17.569ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:42:17.569ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:42:17.570ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:42:17.572ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:42:17.587ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:42:17.587ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:42:17.587ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:42:17.587ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:42:17.594ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:42:17.594ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:42:17.595ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:42:17.598ZINFOtest_delete_snapshot (DataLoader): created built-in services vpc subnets
5612025-04-30T06:42:17.598ZINFOtest_delete_snapshot (DataLoader): created built-in services vpc
5622025-04-30T06:42:17.598ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:42:17.598ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:42:17.603ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:42:17.603ZDEBGtest_delete_snapshot (DataLoader): attempting to create silo users
5662025-04-30T06:42:17.603ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:42:17.603ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:42:17.604ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:42:17.612ZINFOtest_delete_snapshot (DataLoader): created 2 silo users
5702025-04-30T06:42:17.612ZDEBGtest_delete_snapshot (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:42:17.612ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:42:17.612ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:42:17.613ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:42:17.616ZINFOtest_delete_snapshot (DataLoader): created 2 silo user role assignments
5752025-04-30T06:42:17.616ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:42:17.616ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:42:17.617ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:42:17.625ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:42:17.625ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:42:17.625ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:42:17.634ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:42:17.634ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:42:17.634ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:42:17.634ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:42:17.635ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:42:17.637ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:42:17.640ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:42:17.640ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:42:17.640ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:42:17.640ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:42:17.647ZTRACtest_delete_snapshot (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:42:17.647ZTRACtest_delete_snapshot (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:42:17.647ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:42:17.647ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:42:17.647ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:42:17.653ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:42:17.656ZDEBGtest_delete_snapshot (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:42:17.656ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:42:17.656ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:42:17.656ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:42:17.748ZTRACtest_delete_snapshot (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:42:17.748ZDEBGtest_delete_snapshot (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:42:17.749ZDEBGtest_delete_snapshot (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:42:17.758ZINFOtest_delete_snapshot (nexus): populate complete
6052025-04-30T06:42:17.758ZINFOtest_delete_snapshot (nexus): activating background tasks
6062025-04-30T06:42:17.788ZDEBGtest_delete_snapshot (init_with_steps): Running step start_sled1
6072025-04-30T06:42:17.789ZINFOtest_delete_snapshot (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:42:17.846ZINFOtest_delete_snapshot (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:42:17.846ZINFOtest_delete_snapshot (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:42:17.847ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:42:17.847ZDEBGtest_delete_snapshot (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:42:17.847ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:42:17.847ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:42:17.848ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:42:17.848ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6162025-04-30T06:42:17.848ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6172025-04-30T06:42:17.848ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6182025-04-30T06:42:17.848ZWARNtest_delete_snapshot (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6192025-04-30T06:42:17.848ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 105.199\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6202025-04-30T06:42:17.848ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6212025-04-30T06:42:17.848ZDEBGtest_delete_snapshot (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6222025-04-30T06:42:17.848ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
6232025-04-30T06:42:17.848ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6242025-04-30T06:42:17.850ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6252025-04-30T06:42:17.850ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6262025-04-30T06:42:17.850ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6272025-04-30T06:42:17.850ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
6282025-04-30T06:42:17.850ZWARNtest_delete_snapshot (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
6292025-04-30T06:42:17.850ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 49.248\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6302025-04-30T06:42:17.850ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
6312025-04-30T06:42:17.850ZINFOtest_delete_snapshot (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:32:17.850521712 UTC
6322025-04-30T06:42:17.850ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
6332025-04-30T06:42:17.850ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
6342025-04-30T06:42:17.851ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
6352025-04-30T06:42:17.851ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
6362025-04-30T06:42:17.851ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
6372025-04-30T06:42:17.851ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6382025-04-30T06:42:17.851ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
6392025-04-30T06:42:17.851ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6402025-04-30T06:42:17.851ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6412025-04-30T06:42:17.851ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
6422025-04-30T06:42:17.851ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
6432025-04-30T06:42:17.851ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6442025-04-30T06:42:17.852ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6452025-04-30T06:42:17.853ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
6462025-04-30T06:42:17.853ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
6472025-04-30T06:42:17.853ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6482025-04-30T06:42:17.853ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6492025-04-30T06:42:17.853ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
6502025-04-30T06:42:17.853ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
6512025-04-30T06:42:17.853ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
6522025-04-30T06:42:17.853ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6532025-04-30T06:42:17.853ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
6542025-04-30T06:42:17.853ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6552025-04-30T06:42:17.854ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6562025-04-30T06:42:17.854ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
6572025-04-30T06:42:17.854ZWARNtest_delete_snapshot (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
6582025-04-30T06:42:17.854ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 52.485\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6592025-04-30T06:42:17.854ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
6602025-04-30T06:42:17.854ZWARNtest_delete_snapshot (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
6612025-04-30T06:42:17.854ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 16.673\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6622025-04-30T06:42:17.854ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
6632025-04-30T06:42:17.854ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
6642025-04-30T06:42:17.854ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6652025-04-30T06:42:17.854ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
6662025-04-30T06:42:17.854ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
6672025-04-30T06:42:17.855ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
6682025-04-30T06:42:17.855ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
6692025-04-30T06:42:17.855ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
6702025-04-30T06:42:17.855ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6712025-04-30T06:42:17.855ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
6722025-04-30T06:42:17.855ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6732025-04-30T06:42:17.855ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6742025-04-30T06:42:17.855ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
6752025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 19.829\u{b5}s iteration = 1 status = {"error":"task disabled"}
6762025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
6772025-04-30T06:42:17.856ZWARNtest_delete_snapshot (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
6782025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 14.058\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6792025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
6802025-04-30T06:42:17.856ZINFOtest_delete_snapshot (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
6812025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 83.787\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6822025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
6832025-04-30T06:42:17.856ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
6842025-04-30T06:42:17.856ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6852025-04-30T06:42:17.856ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
6862025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6872025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6882025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
6892025-04-30T06:42:17.856ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
6902025-04-30T06:42:17.856ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6912025-04-30T06:42:17.856ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
6922025-04-30T06:42:17.856ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6932025-04-30T06:42:17.857ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6942025-04-30T06:42:17.857ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
6952025-04-30T06:42:17.857ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
6962025-04-30T06:42:17.857ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6972025-04-30T06:42:17.858ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6982025-04-30T06:42:17.858ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
6992025-04-30T06:42:17.858ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7002025-04-30T06:42:17.858ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7012025-04-30T06:42:17.858ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7022025-04-30T06:42:17.859ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7032025-04-30T06:42:17.859ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7042025-04-30T06:42:17.859ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7052025-04-30T06:42:17.859ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7062025-04-30T06:42:17.859ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7072025-04-30T06:42:17.859ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7082025-04-30T06:42:17.859ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7092025-04-30T06:42:17.859ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7102025-04-30T06:42:17.859ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7112025-04-30T06:42:17.860ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7122025-04-30T06:42:17.860ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7132025-04-30T06:42:17.860ZINFOtest_delete_snapshot (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7142025-04-30T06:42:17.860ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = instance_updater elapsed = 297.072\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7152025-04-30T06:42:17.860ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7162025-04-30T06:42:17.860ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7172025-04-30T06:42:17.860ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7182025-04-30T06:42:17.861ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7192025-04-30T06:42:17.861ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7202025-04-30T06:42:17.861ZDEBGtest_delete_snapshot (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7212025-04-30T06:42:17.861ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
7222025-04-30T06:42:17.861ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7232025-04-30T06:42:17.862ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7242025-04-30T06:42:17.862ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
7252025-04-30T06:42:17.862ZINFOtest_delete_snapshot (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
7262025-04-30T06:42:17.862ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
7272025-04-30T06:42:17.862ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7282025-04-30T06:42:17.862ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
7292025-04-30T06:42:17.862ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7302025-04-30T06:42:17.863ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7312025-04-30T06:42:17.863ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
7322025-04-30T06:42:17.863ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
7332025-04-30T06:42:17.863ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
7342025-04-30T06:42:17.871ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
7352025-04-30T06:42:17.871ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
7362025-04-30T06:42:17.872ZDEBGtest_delete_snapshot (SagaRecoverer): listing candidate sagas for recovery
7372025-04-30T06:42:17.872ZTRACtest_delete_snapshot (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
7382025-04-30T06:42:17.872ZDEBGtest_delete_snapshot (SagaRecoverer): roles roles = RoleSet { roles: {} }
7392025-04-30T06:42:17.874ZDEBGtest_delete_snapshot (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
7402025-04-30T06:42:17.874ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
7412025-04-30T06:42:17.874ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
7422025-04-30T06:42:17.874ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
7432025-04-30T06:42:17.877ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
7442025-04-30T06:42:17.877ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
7452025-04-30T06:42:17.877ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
7462025-04-30T06:42:17.877ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7472025-04-30T06:42:17.877ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7482025-04-30T06:42:17.877ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
7492025-04-30T06:42:17.877ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
7502025-04-30T06:42:17.877ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7512025-04-30T06:42:17.877ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
7522025-04-30T06:42:17.877ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
7532025-04-30T06:42:17.878ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
7542025-04-30T06:42:17.878ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7552025-04-30T06:42:17.878ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7562025-04-30T06:42:17.878ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
7572025-04-30T06:42:17.878ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
7582025-04-30T06:42:17.878ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7592025-04-30T06:42:17.878ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
7602025-04-30T06:42:17.878ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
7612025-04-30T06:42:17.878ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
7622025-04-30T06:42:17.878ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7632025-04-30T06:42:17.878ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
7642025-04-30T06:42:17.878ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7652025-04-30T06:42:17.879ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7662025-04-30T06:42:17.879ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
7672025-04-30T06:42:17.879ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
7682025-04-30T06:42:17.879ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7692025-04-30T06:42:17.879ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
7702025-04-30T06:42:17.879ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
7712025-04-30T06:42:17.879ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
7722025-04-30T06:42:17.879ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
7732025-04-30T06:42:17.880ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
7742025-04-30T06:42:17.880ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
7752025-04-30T06:42:17.880ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
7762025-04-30T06:42:17.880ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
7772025-04-30T06:42:17.881ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
7782025-04-30T06:42:17.881ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
7792025-04-30T06:42:17.881ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7802025-04-30T06:42:17.881ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
7812025-04-30T06:42:17.881ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7822025-04-30T06:42:17.882ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7832025-04-30T06:42:17.882ZINFOtest_delete_snapshot (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmppBqSLw/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7842025-04-30T06:42:17.882ZDEBGtest_delete_snapshot (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmppBqSLw/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7852025-04-30T06:42:17.882ZINFOtest_delete_snapshot (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpx1yq0K/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7862025-04-30T06:42:17.882ZDEBGtest_delete_snapshot (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpx1yq0K/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7872025-04-30T06:42:17.938ZDEBGtest_delete_snapshot (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:62141 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7882025-04-30T06:42:17.938ZINFOtest_delete_snapshot (dropshot (Repo Depot)): listening local_addr = [::1]:62141 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7892025-04-30T06:42:17.939ZDEBGtest_delete_snapshot (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:62141 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7902025-04-30T06:42:17.945ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7912025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7922025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7932025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7942025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7952025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7962025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7972025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7982025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7992025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8002025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8012025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8022025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8032025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8042025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8052025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8062025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8072025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8082025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8092025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8102025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8112025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8122025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8132025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8142025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8152025-04-30T06:42:17.947ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8162025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8172025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8182025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8192025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8202025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8212025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8222025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8232025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8242025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8252025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8262025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8272025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8282025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8292025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8302025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8312025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8322025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8332025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8342025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8352025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8362025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8372025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8382025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8392025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8402025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8412025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8422025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8432025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8442025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8452025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8462025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8472025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8482025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8492025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8502025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8512025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8522025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8532025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8542025-04-30T06:42:17.948ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8552025-04-30T06:42:17.949ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8562025-04-30T06:42:17.949ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8572025-04-30T06:42:17.949ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8582025-04-30T06:42:17.949ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:62607 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8592025-04-30T06:42:17.949ZINFOtest_delete_snapshot (dropshot): listening local_addr = [::1]:62607 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8602025-04-30T06:42:17.949ZDEBGtest_delete_snapshot (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:62607 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8612025-04-30T06:42:17.949ZDEBGtest_delete_snapshot (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8622025-04-30T06:42:17.949ZDEBGtest_delete_snapshot (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:35404/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8632025-04-30T06:42:17.950ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 2a 6a 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 *j..........._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:49647 req_id = c8e1103a-e4c4-4fbe-9d26-0a03a82f18e4
8642025-04-30T06:42:17.950ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10858,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49647 req_id = c8e1103a-e4c4-4fbe-9d26-0a03a82f18e4
8652025-04-30T06:42:17.951ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10858, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49647 req_id = c8e1103a-e4c4-4fbe-9d26-0a03a82f18e4
8662025-04-30T06:42:17.951ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 44 87 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 D............_ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64090 req_id = 729892af-9e7c-46f1-9608-0b12d7d51ded
8672025-04-30T06:42:17.951ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17543,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64090 req_id = 729892af-9e7c-46f1-9608-0b12d7d51ded
8682025-04-30T06:42:17.951ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17543, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64090 req_id = 729892af-9e7c-46f1-9608-0b12d7d51ded
8692025-04-30T06:42:17.951ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8702025-04-30T06:42:17.951ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8712025-04-30T06:42:17.951ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8722025-04-30T06:42:17.952ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8732025-04-30T06:42:17.953ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8742025-04-30T06:42:17.953ZINFOtest_delete_snapshot (dropshot_internal): accepted connection local_addr = 127.0.0.1:35404 remote_addr = 127.0.0.1:47370
8752025-04-30T06:42:17.953ZWARNtest_delete_snapshot (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8762025-04-30T06:42:17.953ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 105.539581ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8772025-04-30T06:42:17.953ZWARNtest_delete_snapshot (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8782025-04-30T06:42:17.953ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 103.5365ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8792025-04-30T06:42:17.955ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:47370 req_id = cc082dcb-24d1-4311-b402-1944473ae6fa uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8802025-04-30T06:42:17.957ZINFOtest_delete_snapshot (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8812025-04-30T06:42:17.969ZINFOtest_delete_snapshot (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:32:17.850521712 UTC pruned = {}
8822025-04-30T06:42:17.978ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 128.380119ms iteration = 1 status = {"expiration":"2025-04-30T06:32:17.850521712Z","pruned":[]}
8832025-04-30T06:42:17.979ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8842025-04-30T06:42:17.982ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 01 bf 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59908 req_id = 294fa17d-16cf-4ca7-a8b9-1bba550b9c25
8852025-04-30T06:42:17.982ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 447,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59908 req_id = 294fa17d-16cf-4ca7-a8b9-1bba550b9c25
8862025-04-30T06:42:17.983ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 447, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59908 req_id = 294fa17d-16cf-4ca7-a8b9-1bba550b9c25
8872025-04-30T06:42:17.983ZERROtest_delete_snapshot (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8882025-04-30T06:42:17.983ZERROtest_delete_snapshot (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8892025-04-30T06:42:17.983ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 132.050132ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8902025-04-30T06:42:17.985ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8912025-04-30T06:42:17.991ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8922025-04-30T06:42:17.991ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
8932025-04-30T06:42:17.991ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8942025-04-30T06:42:17.991ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8952025-04-30T06:42:17.992ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8962025-04-30T06:42:17.996ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: f5 e0 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63241 req_id = 26e6c918-aab4-4065-b939-9321337bccc6
8972025-04-30T06:42:17.997ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62944,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63241 req_id = 26e6c918-aab4-4065-b939-9321337bccc6
8982025-04-30T06:42:17.997ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62944, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63241 req_id = 26e6c918-aab4-4065-b939-9321337bccc6
8992025-04-30T06:42:17.997ZERROtest_delete_snapshot (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9002025-04-30T06:42:17.997ZERROtest_delete_snapshot (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9012025-04-30T06:42:17.997ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 144.810395ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9022025-04-30T06:42:18.008ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9032025-04-30T06:42:18.038ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9042025-04-30T06:42:18.038ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9052025-04-30T06:42:18.038ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9062025-04-30T06:42:18.038ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9072025-04-30T06:42:18.073ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 219.381253ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
9082025-04-30T06:42:18.073ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9092025-04-30T06:42:18.077ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9102025-04-30T06:42:18.077ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9112025-04-30T06:42:18.077ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9122025-04-30T06:42:18.080ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9132025-04-30T06:42:18.080ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9142025-04-30T06:42:18.086ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9152025-04-30T06:42:18.086ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9162025-04-30T06:42:18.086ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9172025-04-30T06:42:18.086ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9182025-04-30T06:42:18.087ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9192025-04-30T06:42:18.100ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9202025-04-30T06:42:18.100ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9212025-04-30T06:42:18.100ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9222025-04-30T06:42:18.100ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9232025-04-30T06:42:18.115ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9242025-04-30T06:42:18.120ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9252025-04-30T06:42:18.121ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9262025-04-30T06:42:18.122ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9272025-04-30T06:42:18.123ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9282025-04-30T06:42:18.251ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9292025-04-30T06:42:18.255ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9302025-04-30T06:42:18.256ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9312025-04-30T06:42:18.259ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9322025-04-30T06:42:18.260ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9332025-04-30T06:42:18.260ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9342025-04-30T06:42:18.260ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9352025-04-30T06:42:18.261ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9362025-04-30T06:42:18.327ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9372025-04-30T06:42:18.327ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9382025-04-30T06:42:18.328ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9392025-04-30T06:42:18.347ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9402025-04-30T06:42:18.347ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9412025-04-30T06:42:18.348ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9422025-04-30T06:42:18.350ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9432025-04-30T06:42:18.354ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9442025-04-30T06:42:18.354ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9452025-04-30T06:42:18.354ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9462025-04-30T06:42:18.354ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9472025-04-30T06:42:18.354ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9482025-04-30T06:42:18.354ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9492025-04-30T06:42:18.367ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 492.847016ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9502025-04-30T06:42:18.427ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9512025-04-30T06:42:18.432ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9522025-04-30T06:42:18.432ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9532025-04-30T06:42:18.432ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9542025-04-30T06:42:18.432ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9552025-04-30T06:42:18.432ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9562025-04-30T06:42:18.432ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9572025-04-30T06:42:18.487ZTRACtest_delete_snapshot (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9582025-04-30T06:42:18.487ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9592025-04-30T06:42:18.487ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9602025-04-30T06:42:18.488ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9612025-04-30T06:42:18.488ZINFOtest_delete_snapshot (SagaRecoverer): listed in-progress sagas count = 0
9622025-04-30T06:42:18.488ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 617.155001ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9632025-04-30T06:42:18.489ZDEBGtest_delete_snapshot (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9642025-04-30T06:42:18.489ZINFOtest_delete_snapshot (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9652025-04-30T06:42:18.489ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 626.546143ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9662025-04-30T06:42:18.503ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9672025-04-30T06:42:18.511ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9682025-04-30T06:42:18.531ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9692025-04-30T06:42:18.537ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9702025-04-30T06:42:18.575ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 695.146544ms iteration = 1 status = {"by_rx":{},"error":null}
9712025-04-30T06:42:18.696ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9722025-04-30T06:42:18.696ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9732025-04-30T06:42:18.697ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9742025-04-30T06:42:18.697ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9752025-04-30T06:42:18.697ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9762025-04-30T06:42:18.704ZWARNtest_delete_snapshot (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
9772025-04-30T06:42:18.704ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 850.507341ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9782025-04-30T06:42:18.708ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 831.247356ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9792025-04-30T06:42:18.709ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9802025-04-30T06:42:18.709ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9812025-04-30T06:42:18.709ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9822025-04-30T06:42:18.715ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
9832025-04-30T06:42:18.715ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9842025-04-30T06:42:18.715ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9852025-04-30T06:42:18.736ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 857.57357ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
9862025-04-30T06:42:18.741ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9872025-04-30T06:42:18.741ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9882025-04-30T06:42:18.741ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9892025-04-30T06:42:18.752ZWARNtest_delete_snapshot (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
9902025-04-30T06:42:18.752ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 903.658701ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
9912025-04-30T06:42:18.752ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 876.840568ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
9922025-04-30T06:42:18.756ZWARNtest_delete_snapshot (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
9932025-04-30T06:42:18.756ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 909.498356ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
9942025-04-30T06:42:18.791ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
9952025-04-30T06:42:18.791ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
9962025-04-30T06:42:18.792ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c3 0a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57827 req_id = 5cb76803-e74e-4672-8a4e-0ff4f7088e78
9972025-04-30T06:42:18.792ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49930,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57827 req_id = 5cb76803-e74e-4672-8a4e-0ff4f7088e78
9982025-04-30T06:42:18.792ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49930, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57827 req_id = 5cb76803-e74e-4672-8a4e-0ff4f7088e78
9992025-04-30T06:42:18.793ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
10002025-04-30T06:42:18.793ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 939.030296ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
10012025-04-30T06:42:18.846ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 990.004038ms iteration = 1 status = {}
10022025-04-30T06:42:18.846ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
10032025-04-30T06:42:18.846ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 990.856277ms iteration = 1 status = {"error":"inventory collection is None"}
10042025-04-30T06:42:18.850ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10052025-04-30T06:42:18.850ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10062025-04-30T06:42:18.851ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10072025-04-30T06:42:18.857ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10082025-04-30T06:42:18.857ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10092025-04-30T06:42:18.857ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10102025-04-30T06:42:18.857ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10112025-04-30T06:42:18.857ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10122025-04-30T06:42:18.857ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10132025-04-30T06:42:18.857ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 902616 local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:47370 req_id = cc082dcb-24d1-4311-b402-1944473ae6fa response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10142025-04-30T06:42:18.858ZDEBGtest_delete_snapshot (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:35404/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "cc082dcb-24d1-4311-b402-1944473ae6fa", "date": "Wed, 30 Apr 2025 06:42:17 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10152025-04-30T06:42:18.858ZDEBGtest_delete_snapshot (init_with_steps): Running step start_crucible_pantry
10162025-04-30T06:42:18.859ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10172025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10182025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10192025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10202025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10212025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10222025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10232025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10242025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10252025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10262025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): registered endpoint kind = pantry local_addr = [::1]:33598 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10272025-04-30T06:42:18.860ZINFOtest_delete_snapshot (dropshot): listening kind = pantry local_addr = [::1]:33598 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10282025-04-30T06:42:18.860ZDEBGtest_delete_snapshot (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:33598 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10292025-04-30T06:42:18.860ZINFOtest_delete_snapshot (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:33598 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10302025-04-30T06:42:18.861ZDEBGtest_delete_snapshot (init_with_steps): Running step populate_internal_dns
10312025-04-30T06:42:18.861ZDEBGtest_delete_snapshot: Populating Internal DNS
10322025-04-30T06:42:18.986ZINFOtest_delete_snapshot: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:42:18.985950821Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "ca761d4f-8fbb-4386-a3ef-90926f06ae50.host": [ Aaaa( ::1, ), ], "f08867ef-19ac-4764-90de-9146c69b8f3f.host": [ Aaaa( ::1, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "929a31e2-4c7b-4c43-a7ce-74d8ff3c0147.host.control-plane.oxide.internal", }, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 65050, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 34703, target: "19d6143b-d959-44e3-88ec-1900a62c7ae5.host.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "929a31e2-4c7b-4c43-a7ce-74d8ff3c0147.host.control-plane.oxide.internal", }, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 35772, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 41697, target: "ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal", }, ), ], "19d6143b-d959-44e3-88ec-1900a62c7ae5.host": [ Aaaa( ::1, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 62141, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 35404, target: "de3c7c83-8252-43f4-ba2e-cbd92004c4b0.host.control-plane.oxide.internal", }, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 51007, target: "f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal", }, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 44357, target: "929a31e2-4c7b-4c43-a7ce-74d8ff3c0147.host.control-plane.oxide.internal", }, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 33598, target: "7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host.control-plane.oxide.internal", }, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "929a31e2-4c7b-4c43-a7ce-74d8ff3c0147.host.control-plane.oxide.internal", }, ), ], "de3c7c83-8252-43f4-ba2e-cbd92004c4b0.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 50898, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "929a31e2-4c7b-4c43-a7ce-74d8ff3c0147.host": [ Aaaa( ::1, ), ], "7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host": [ Aaaa( ::1, ), ], }, }, ], }
10332025-04-30T06:42:19.005ZDEBGtest_delete_snapshot: client request body = Some(Body) method = PUT uri = http://[::1]:51007/config
10342025-04-30T06:42:19.019ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10352025-04-30T06:42:19.020ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10362025-04-30T06:42:19.020ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10372025-04-30T06:42:19.020ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10382025-04-30T06:42:19.020ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10392025-04-30T06:42:19.021ZINFOtest_delete_snapshot (http): accepted connection kind = dns local_addr = [::1]:51007 remote_addr = [::1]:56297
10402025-04-30T06:42:19.022ZTRACtest_delete_snapshot (http): incoming request kind = dns local_addr = [::1]:51007 method = PUT remote_addr = [::1]:56297 req_id = c7bf873e-880a-4c9b-aecc-5b2fed74e975 uri = /config
10412025-04-30T06:42:19.023ZINFOtest_delete_snapshot (store): attempting generation update new_generation = 1 req_id = c7bf873e-880a-4c9b-aecc-5b2fed74e975
10422025-04-30T06:42:19.023ZINFOtest_delete_snapshot (store): pruning trees for generations newer than 0
10432025-04-30T06:42:19.023ZDEBGtest_delete_snapshot (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10442025-04-30T06:42:19.024ZDEBGtest_delete_snapshot (store): updating current config
10452025-04-30T06:42:19.024ZDEBGtest_delete_snapshot (store): flushing default tree
10462025-04-30T06:42:19.024ZINFOtest_delete_snapshot (store): pruning trees for generations older than 1 keep = 3
10472025-04-30T06:42:19.024ZINFOtest_delete_snapshot (store): updated generation new_generation = 1 req_id = c7bf873e-880a-4c9b-aecc-5b2fed74e975
10482025-04-30T06:42:19.025ZINFOtest_delete_snapshot (http): request completed kind = dns latency_us = 2721 local_addr = [::1]:51007 method = PUT remote_addr = [::1]:56297 req_id = c7bf873e-880a-4c9b-aecc-5b2fed74e975 response_code = 204 uri = /config
10492025-04-30T06:42:19.025ZDEBGtest_delete_snapshot: client response result = Ok(Response { url: "http://[::1]:51007/config", status: 204, headers: {"x-request-id": "c7bf873e-880a-4c9b-aecc-5b2fed74e975", "date": "Wed, 30 Apr 2025 06:42:19 GMT"} })
10502025-04-30T06:42:19.026ZDEBGtest_delete_snapshot (init_with_steps): Running step configure_sled_agents
10512025-04-30T06:42:19.080ZDEBGtest_delete_snapshot: client request body = Some(Body) method = PUT uri = http://[::1]:62607/omicron-config
10522025-04-30T06:42:19.081ZINFOtest_delete_snapshot (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10532025-04-30T06:42:19.081ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 1.221911519s iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10542025-04-30T06:42:19.081ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = region_replacement elapsed = 1.223493562s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10552025-04-30T06:42:19.082ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:38977 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10562025-04-30T06:42:19.086ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = PUT remote_addr = [::1]:38977 req_id = 3fa8e74a-24d0-4ac7-9b11-26299a3e09a9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10572025-04-30T06:42:19.089ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 3666 local_addr = [::1]:62607 method = PUT remote_addr = [::1]:38977 req_id = 3fa8e74a-24d0-4ac7-9b11-26299a3e09a9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10582025-04-30T06:42:19.091ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10592025-04-30T06:42:19.094ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10602025-04-30T06:42:19.094ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10612025-04-30T06:42:19.094ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10622025-04-30T06:42:19.095ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10632025-04-30T06:42:19.095ZDEBGtest_delete_snapshot: client response result = Ok(Response { url: "http://[::1]:62607/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fa8e74a-24d0-4ac7-9b11-26299a3e09a9", "content-length": "26", "date": "Wed, 30 Apr 2025 06:42:19 GMT"} })
10642025-04-30T06:42:19.095ZDEBGtest_delete_snapshot: client request body = Some(Body) method = PUT uri = http://[::1]:62607/network-bootstore-config
10652025-04-30T06:42:19.100ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = PUT remote_addr = [::1]:38977 req_id = 6f972b00-ae4e-450e-8688-5c37f329fa03 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10662025-04-30T06:42:19.100ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 615 local_addr = [::1]:62607 method = PUT remote_addr = [::1]:38977 req_id = 6f972b00-ae4e-450e-8688-5c37f329fa03 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10672025-04-30T06:42:19.103ZDEBGtest_delete_snapshot: client response result = Ok(Response { url: "http://[::1]:62607/network-bootstore-config", status: 204, headers: {"x-request-id": "6f972b00-ae4e-450e-8688-5c37f329fa03", "date": "Wed, 30 Apr 2025 06:42:19 GMT"} })
10682025-04-30T06:42:19.104ZDEBGtest_delete_snapshot (init_with_steps): Running step start_nexus_external
10692025-04-30T06:42:19.104ZDEBGtest_delete_snapshot: Starting Nexus (external API)
10702025-04-30T06:42:22.017ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
10712025-04-30T06:42:22.017ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10722025-04-30T06:42:22.017ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
10732025-04-30T06:42:22.017ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
10742025-04-30T06:42:22.018ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10752025-04-30T06:42:22.020ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.160957836s iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10762025-04-30T06:42:22.020ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10772025-04-30T06:42:22.024ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10782025-04-30T06:42:22.024ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10792025-04-30T06:42:22.024ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10802025-04-30T06:42:22.025ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10812025-04-30T06:42:22.035ZTRACtest_delete_snapshot (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10822025-04-30T06:42:22.036ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10832025-04-30T06:42:22.036ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10842025-04-30T06:42:22.037ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10852025-04-30T06:42:22.038ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10862025-04-30T06:42:22.047ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10872025-04-30T06:42:22.047ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10882025-04-30T06:42:22.047ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10892025-04-30T06:42:22.048ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10902025-04-30T06:42:22.052ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 4.176903306s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10912025-04-30T06:42:22.053ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10922025-04-30T06:42:22.054ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10932025-04-30T06:42:22.054ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10942025-04-30T06:42:22.054ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10952025-04-30T06:42:22.056ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10962025-04-30T06:42:22.056ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10972025-04-30T06:42:22.062ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
10982025-04-30T06:42:22.062ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10992025-04-30T06:42:22.062ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
11002025-04-30T06:42:22.063ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
11012025-04-30T06:42:22.112ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11022025-04-30T06:42:22.112ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11032025-04-30T06:42:22.112ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11042025-04-30T06:42:22.112ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11052025-04-30T06:42:22.113ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11062025-04-30T06:42:22.114ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
11072025-04-30T06:42:22.114ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11082025-04-30T06:42:22.115ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11092025-04-30T06:42:22.152ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11102025-04-30T06:42:22.156ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11112025-04-30T06:42:22.156ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
11122025-04-30T06:42:22.156ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11132025-04-30T06:42:22.157ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11142025-04-30T06:42:22.167ZTRACtest_delete_snapshot (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
11152025-04-30T06:42:22.167ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 4.287775943s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
11162025-04-30T06:42:22.245ZINFOtest_delete_snapshot (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
11172025-04-30T06:42:22.245ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:62607/vpc-routes
11182025-04-30T06:42:22.245ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11192025-04-30T06:42:22.245ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11202025-04-30T06:42:22.245ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11212025-04-30T06:42:22.246ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11222025-04-30T06:42:22.246ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11232025-04-30T06:42:22.247ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11242025-04-30T06:42:22.247ZTRACtest_delete_snapshot (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
11252025-04-30T06:42:22.247ZINFOtest_delete_snapshot (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11262025-04-30T06:42:22.247ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 4.386886244s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11272025-04-30T06:42:22.355ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11282025-04-30T06:42:22.355ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11292025-04-30T06:42:22.355ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11302025-04-30T06:42:22.355ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11312025-04-30T06:42:22.355ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11322025-04-30T06:42:22.356ZWARNtest_delete_snapshot (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
11332025-04-30T06:42:22.356ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 4.504783657s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
11342025-04-30T06:42:22.357ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11352025-04-30T06:42:22.384ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11362025-04-30T06:42:22.385ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11372025-04-30T06:42:22.388ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11382025-04-30T06:42:22.388ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11392025-04-30T06:42:22.388ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11402025-04-30T06:42:22.393ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11412025-04-30T06:42:22.394ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11422025-04-30T06:42:22.397ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11432025-04-30T06:42:22.398ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11442025-04-30T06:42:22.398ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
11452025-04-30T06:42:22.399ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11462025-04-30T06:42:22.404ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:60788 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11472025-04-30T06:42:22.429ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 4.552753424s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11482025-04-30T06:42:22.476ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:60788 req_id = 25d554f8-92d8-46c3-adb9-c81b6bf26c53 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11492025-04-30T06:42:22.476ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d7 b3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52234 req_id = 2bdd4805-a2cd-4c04-a7bf-9318abcbf0fb
11502025-04-30T06:42:22.476ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55219,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52234 req_id = 2bdd4805-a2cd-4c04-a7bf-9318abcbf0fb
11512025-04-30T06:42:22.477ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
11522025-04-30T06:42:22.477ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11532025-04-30T06:42:22.478ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52234 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2bdd4805-a2cd-4c04-a7bf-9318abcbf0fb
11542025-04-30T06:42:22.479ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 2584 local_addr = [::1]:62607 method = GET remote_addr = [::1]:60788 req_id = 25d554f8-92d8-46c3-adb9-c81b6bf26c53 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11552025-04-30T06:42:22.492ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11562025-04-30T06:42:22.492ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11572025-04-30T06:42:22.492ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11582025-04-30T06:42:22.493ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11592025-04-30T06:42:22.493ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11602025-04-30T06:42:22.498ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11612025-04-30T06:42:22.498ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11622025-04-30T06:42:22.498ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11632025-04-30T06:42:22.500ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11642025-04-30T06:42:22.501ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11652025-04-30T06:42:22.504ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11662025-04-30T06:42:22.504ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11672025-04-30T06:42:22.504ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11682025-04-30T06:42:22.505ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11692025-04-30T06:42:22.506ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4207, tv_nsec: 305808966 } })
11702025-04-30T06:42:22.507ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
11712025-04-30T06:42:22.509ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:62607/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "25d554f8-92d8-46c3-adb9-c81b6bf26c53", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:22 GMT"} })
11722025-04-30T06:42:22.509ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
11732025-04-30T06:42:22.509ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11742025-04-30T06:42:22.510ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11752025-04-30T06:42:22.523ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 11 9d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57137 req_id = e03fa3de-ccd7-431c-aba6-23d93c42957e
11762025-04-30T06:42:22.524ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4509,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57137 req_id = e03fa3de-ccd7-431c-aba6-23d93c42957e
11772025-04-30T06:42:22.524ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11782025-04-30T06:42:22.524ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57137 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e03fa3de-ccd7-431c-aba6-23d93c42957e
11792025-04-30T06:42:22.642ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
11802025-04-30T06:42:22.642ZINFOtest_delete_snapshot (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
11812025-04-30T06:42:22.643ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 4.787764157s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11822025-04-30T06:42:22.643ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11832025-04-30T06:42:22.643ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
11842025-04-30T06:42:22.644ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11852025-04-30T06:42:22.661ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11862025-04-30T06:42:22.681ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11872025-04-30T06:42:22.681ZTRACtest_delete_snapshot (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11882025-04-30T06:42:22.681ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11892025-04-30T06:42:22.681ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11902025-04-30T06:42:22.681ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11912025-04-30T06:42:22.681ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11922025-04-30T06:42:22.682ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11932025-04-30T06:42:22.683ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11942025-04-30T06:42:22.683ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11952025-04-30T06:42:22.683ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11962025-04-30T06:42:22.683ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11972025-04-30T06:42:22.684ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11982025-04-30T06:42:22.780ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:62607/v2p
11992025-04-30T06:42:22.782ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:37346 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12002025-04-30T06:42:22.782ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ff b5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52649 req_id = a281321a-d691-4ae7-993e-61cfe4f0d178
12012025-04-30T06:42:22.782ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65461,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52649 req_id = a281321a-d691-4ae7-993e-61cfe4f0d178
12022025-04-30T06:42:22.783ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
12032025-04-30T06:42:22.783ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65461, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52649 req_id = a281321a-d691-4ae7-993e-61cfe4f0d178
12042025-04-30T06:42:22.790ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
12052025-04-30T06:42:22.790ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
12062025-04-30T06:42:22.790ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
12072025-04-30T06:42:22.791ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:37346 req_id = c50dc9fe-e23e-4d21-ae80-a7ef11c6adc5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12082025-04-30T06:42:22.792ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1761 local_addr = [::1]:62607 method = GET remote_addr = [::1]:37346 req_id = c50dc9fe-e23e-4d21-ae80-a7ef11c6adc5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12092025-04-30T06:42:22.805ZINFOtest_delete_snapshot (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
12102025-04-30T06:42:22.805ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:62607/eip-gateways
12112025-04-30T06:42:22.809ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = PUT remote_addr = [::1]:60788 req_id = 3a3fd179-c5bc-4c37-a11a-f3272d6249cb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12122025-04-30T06:42:22.809ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:62607/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c50dc9fe-e23e-4d21-ae80-a7ef11c6adc5", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:22 GMT"} })
12132025-04-30T06:42:22.810ZINFOtest_delete_snapshot (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12142025-04-30T06:42:22.810ZINFOtest_delete_snapshot (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12152025-04-30T06:42:22.810ZINFOtest_delete_snapshot (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12162025-04-30T06:42:22.812ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 4.955369129s iteration = 1 status = {}
12172025-04-30T06:42:22.812ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 3580 local_addr = [::1]:62607 method = PUT remote_addr = [::1]:60788 req_id = 3a3fd179-c5bc-4c37-a11a-f3272d6249cb response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12182025-04-30T06:42:22.813ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12192025-04-30T06:42:22.817ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12202025-04-30T06:42:22.817ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12212025-04-30T06:42:22.818ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12222025-04-30T06:42:22.818ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12232025-04-30T06:42:22.819ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12242025-04-30T06:42:22.819ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12252025-04-30T06:42:22.819ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12262025-04-30T06:42:22.820ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12272025-04-30T06:42:22.820ZINFOtest_delete_snapshot (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
12282025-04-30T06:42:22.821ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12292025-04-30T06:42:22.821ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
12302025-04-30T06:42:22.821ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12312025-04-30T06:42:22.827ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:62607/eip-gateways", status: 204, headers: {"x-request-id": "3a3fd179-c5bc-4c37-a11a-f3272d6249cb", "date": "Wed, 30 Apr 2025 06:42:22 GMT"} })
12322025-04-30T06:42:22.829ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 4.967147917s iteration = 1 status = {}
12332025-04-30T06:42:22.830ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:62607/artifacts-config
12342025-04-30T06:42:22.832ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:58352 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12352025-04-30T06:42:22.832ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12362025-04-30T06:42:22.833ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12372025-04-30T06:42:22.833ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12382025-04-30T06:42:22.833ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12392025-04-30T06:42:22.834ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12402025-04-30T06:42:22.834ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12412025-04-30T06:42:22.835ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12422025-04-30T06:42:22.835ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
12432025-04-30T06:42:22.835ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12442025-04-30T06:42:22.836ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = PUT remote_addr = [::1]:58352 req_id = 7efe08ad-201d-48cc-b41b-17599f84c419 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12452025-04-30T06:42:22.841ZINFOtest_delete_snapshot (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmppBqSLw/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12462025-04-30T06:42:22.841ZDEBGtest_delete_snapshot (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmppBqSLw/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12472025-04-30T06:42:22.841ZINFOtest_delete_snapshot (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpx1yq0K/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12482025-04-30T06:42:22.841ZDEBGtest_delete_snapshot (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpx1yq0K/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12492025-04-30T06:42:22.841ZINFOtest_delete_snapshot (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmppBqSLw/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12502025-04-30T06:42:22.842ZINFOtest_delete_snapshot (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpx1yq0K/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12512025-04-30T06:42:22.844ZINFOtest_delete_snapshot (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12522025-04-30T06:42:22.845ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 8983 local_addr = [::1]:62607 method = PUT remote_addr = [::1]:58352 req_id = 7efe08ad-201d-48cc-b41b-17599f84c419 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12532025-04-30T06:42:22.845ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12542025-04-30T06:42:22.852ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12552025-04-30T06:42:22.852ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12562025-04-30T06:42:22.852ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12572025-04-30T06:42:22.852ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12582025-04-30T06:42:22.862ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:62607/artifacts-config", status: 204, headers: {"x-request-id": "7efe08ad-201d-48cc-b41b-17599f84c419", "date": "Wed, 30 Apr 2025 06:42:22 GMT"} })
12592025-04-30T06:42:22.862ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:62607/artifacts
12602025-04-30T06:42:22.864ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12612025-04-30T06:42:22.869ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12622025-04-30T06:42:22.869ZDEBGtest_delete_snapshot (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12632025-04-30T06:42:22.869ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12642025-04-30T06:42:22.869ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
12652025-04-30T06:42:22.870ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12662025-04-30T06:42:22.871ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12672025-04-30T06:42:22.871ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12682025-04-30T06:42:22.871ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12692025-04-30T06:42:22.873ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:58352 req_id = 51b1c516-1a08-404f-9622-ad6555f85ae1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12702025-04-30T06:42:22.873ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcbf0069-3866-4730-be29-764abd6ac068", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:22 GMT"} })
12712025-04-30T06:42:22.874ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
12722025-04-30T06:42:22.887ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 14143 local_addr = [::1]:62607 method = GET remote_addr = [::1]:58352 req_id = 51b1c516-1a08-404f-9622-ad6555f85ae1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12732025-04-30T06:42:22.892ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12742025-04-30T06:42:22.892ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12752025-04-30T06:42:22.893ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12762025-04-30T06:42:22.898ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:62607/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "51b1c516-1a08-404f-9622-ad6555f85ae1", "content-length": "26", "date": "Wed, 30 Apr 2025 06:42:22 GMT"} })
12772025-04-30T06:42:22.898ZINFOtest_delete_snapshot (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:62607
12782025-04-30T06:42:22.904ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 5.02671962s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:42:22.862218640Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:42:22.898784686Z"}]}
12792025-04-30T06:42:22.906ZINFOtest_delete_snapshot (InternalApi): Using static configuration for external switchports
12802025-04-30T06:42:22.908ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12812025-04-30T06:42:22.908ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
12822025-04-30T06:42:22.909ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12832025-04-30T06:42:22.910ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12842025-04-30T06:42:22.910ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12852025-04-30T06:42:22.911ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12862025-04-30T06:42:22.916ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12872025-04-30T06:42:22.916ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12882025-04-30T06:42:22.916ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12892025-04-30T06:42:22.916ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12902025-04-30T06:42:22.917ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12912025-04-30T06:42:22.923ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12922025-04-30T06:42:22.923ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12932025-04-30T06:42:22.923ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12942025-04-30T06:42:22.924ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12952025-04-30T06:42:22.930ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12962025-04-30T06:42:22.930ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12972025-04-30T06:42:22.930ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12982025-04-30T06:42:22.931ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12992025-04-30T06:42:22.947ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13002025-04-30T06:42:22.955ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13012025-04-30T06:42:22.955ZTRACtest_delete_snapshot (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13022025-04-30T06:42:22.955ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13032025-04-30T06:42:22.955ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13042025-04-30T06:42:22.956ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13052025-04-30T06:42:22.958ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43d21f25-a360-411c-ad72-c856aba51844", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:22 GMT"} })
13062025-04-30T06:42:22.958ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
13072025-04-30T06:42:22.959ZINFOtest_delete_snapshot (InternalApi): Recording Rack Network Configuration
13082025-04-30T06:42:22.959ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13092025-04-30T06:42:22.959ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
13102025-04-30T06:42:22.959ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13112025-04-30T06:42:22.962ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13122025-04-30T06:42:22.962ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13132025-04-30T06:42:22.962ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13142025-04-30T06:42:22.966ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13152025-04-30T06:42:22.966ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13162025-04-30T06:42:22.966ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13172025-04-30T06:42:22.971ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13182025-04-30T06:42:22.971ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13192025-04-30T06:42:22.972ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13202025-04-30T06:42:22.981ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13212025-04-30T06:42:22.982ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13222025-04-30T06:42:22.982ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13232025-04-30T06:42:22.982ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13242025-04-30T06:42:22.982ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13252025-04-30T06:42:22.991ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:42:22.991ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:42:22.991ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:42:22.992ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:42:22.997ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13302025-04-30T06:42:22.998ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13312025-04-30T06:42:22.998ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13322025-04-30T06:42:22.998ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13332025-04-30T06:42:23.014ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13342025-04-30T06:42:23.025ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13352025-04-30T06:42:23.026ZTRACtest_delete_snapshot (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13362025-04-30T06:42:23.026ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13372025-04-30T06:42:23.026ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13382025-04-30T06:42:23.026ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13392025-04-30T06:42:23.026ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13402025-04-30T06:42:23.027ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13412025-04-30T06:42:23.077ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f90f0c54-dd7f-4590-84be-fe43dd9c04c7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:22 GMT"} })
13422025-04-30T06:42:23.078ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
13432025-04-30T06:42:23.078ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13442025-04-30T06:42:23.078ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13452025-04-30T06:42:23.078ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13462025-04-30T06:42:23.080ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13472025-04-30T06:42:23.097ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13482025-04-30T06:42:23.097ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13492025-04-30T06:42:23.097ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13502025-04-30T06:42:23.098ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13512025-04-30T06:42:23.100ZTRACtest_delete_snapshot (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
13522025-04-30T06:42:23.100ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13532025-04-30T06:42:23.100ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13542025-04-30T06:42:23.100ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
13552025-04-30T06:42:23.101ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13562025-04-30T06:42:23.107ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13572025-04-30T06:42:23.128ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13582025-04-30T06:42:23.128ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13592025-04-30T06:42:23.128ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13602025-04-30T06:42:23.129ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13612025-04-30T06:42:23.131ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13622025-04-30T06:42:23.134ZDEBGtest_delete_snapshot (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
13632025-04-30T06:42:23.134ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13642025-04-30T06:42:23.134ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
13652025-04-30T06:42:23.134ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13662025-04-30T06:42:23.138ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13672025-04-30T06:42:23.138ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13682025-04-30T06:42:23.138ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13692025-04-30T06:42:23.142ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13702025-04-30T06:42:23.142ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13712025-04-30T06:42:23.142ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13722025-04-30T06:42:23.143ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 4aa607f4-d875-4ed4-a528-549fc5d65679, lookup_type: ByName("oxide-service-pool") }
13732025-04-30T06:42:23.143ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13742025-04-30T06:42:23.143ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13752025-04-30T06:42:23.143ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
13762025-04-30T06:42:23.148ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13772025-04-30T06:42:23.159ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13782025-04-30T06:42:23.159ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13792025-04-30T06:42:23.176ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13802025-04-30T06:42:23.181ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13812025-04-30T06:42:23.181ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13822025-04-30T06:42:23.181ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13832025-04-30T06:42:23.203ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9641eb61-132d-4a55-9040-732503d61815", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
13842025-04-30T06:42:23.203ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
13852025-04-30T06:42:23.236ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13862025-04-30T06:42:23.237ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13872025-04-30T06:42:23.237ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13882025-04-30T06:42:23.237ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13892025-04-30T06:42:23.238ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13902025-04-30T06:42:23.239ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92e36d91-970f-4d62-9c36-5a6f57eca678", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
13912025-04-30T06:42:23.239ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
13922025-04-30T06:42:23.291ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13932025-04-30T06:42:23.302ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 4aa607f4-d875-4ed4-a528-549fc5d65679, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13942025-04-30T06:42:23.302ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13952025-04-30T06:42:23.302ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
13962025-04-30T06:42:23.303ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13972025-04-30T06:42:23.304ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13982025-04-30T06:42:23.304ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13992025-04-30T06:42:23.304ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14002025-04-30T06:42:23.304ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14012025-04-30T06:42:23.328ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14022025-04-30T06:42:23.328ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14032025-04-30T06:42:23.329ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14042025-04-30T06:42:23.329ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14052025-04-30T06:42:23.331ZTRACtest_delete_snapshot (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 4aa607f4-d875-4ed4-a528-549fc5d65679, lookup_type: ByName("oxide-service-pool") }
14062025-04-30T06:42:23.331ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14072025-04-30T06:42:23.331ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14082025-04-30T06:42:23.331ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
14092025-04-30T06:42:23.332ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14102025-04-30T06:42:23.333ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14112025-04-30T06:42:23.346ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14122025-04-30T06:42:23.346ZTRACtest_delete_snapshot (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
14132025-04-30T06:42:23.346ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14142025-04-30T06:42:23.346ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14152025-04-30T06:42:23.346ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14162025-04-30T06:42:23.346ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14172025-04-30T06:42:23.346ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14182025-04-30T06:42:23.352ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14192025-04-30T06:42:23.358ZDEBGtest_delete_snapshot (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 4aa607f4-d875-4ed4-a528-549fc5d65679, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14202025-04-30T06:42:23.358ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14212025-04-30T06:42:23.358ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14222025-04-30T06:42:23.358ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14232025-04-30T06:42:23.368ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14242025-04-30T06:42:23.370ZTRACtest_delete_snapshot (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 4aa607f4-d875-4ed4-a528-549fc5d65679, lookup_type: ByName("oxide-service-pool") }
14252025-04-30T06:42:23.370ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14262025-04-30T06:42:23.371ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14272025-04-30T06:42:23.371ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
14282025-04-30T06:42:23.372ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14292025-04-30T06:42:23.374ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14302025-04-30T06:42:23.374ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14312025-04-30T06:42:23.374ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14322025-04-30T06:42:23.375ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14332025-04-30T06:42:23.377ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f5dba8e-c941-4a3a-b251-0e9b23113cad", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14342025-04-30T06:42:23.378ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
14352025-04-30T06:42:23.378ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14362025-04-30T06:42:23.386ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14372025-04-30T06:42:23.386ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14382025-04-30T06:42:23.386ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14392025-04-30T06:42:23.387ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14402025-04-30T06:42:23.388ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14412025-04-30T06:42:23.397ZDEBGtest_delete_snapshot (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 4aa607f4-d875-4ed4-a528-549fc5d65679, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14422025-04-30T06:42:23.410ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14432025-04-30T06:42:23.410ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14442025-04-30T06:42:23.410ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14452025-04-30T06:42:23.410ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
14462025-04-30T06:42:23.422ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14472025-04-30T06:42:23.422ZDEBGtest_delete_snapshot (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
14482025-04-30T06:42:23.422ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14492025-04-30T06:42:23.422ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14502025-04-30T06:42:23.422ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14512025-04-30T06:42:23.422ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14522025-04-30T06:42:23.422ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14532025-04-30T06:42:23.422ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14542025-04-30T06:42:23.422ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14552025-04-30T06:42:23.422ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "784280c0-7e90-48c1-ba9a-d86342bd9529", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14562025-04-30T06:42:23.422ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
14572025-04-30T06:42:23.427ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cdfc815-5d33-4a2b-a0b8-eac664b97608", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14582025-04-30T06:42:23.427ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
14592025-04-30T06:42:23.430ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14602025-04-30T06:42:23.434ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14612025-04-30T06:42:23.434ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14622025-04-30T06:42:23.434ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14632025-04-30T06:42:23.434ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14642025-04-30T06:42:23.438ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97244667-7cac-4f7d-859f-7019ecffaa5d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14652025-04-30T06:42:23.438ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
14662025-04-30T06:42:23.440ZTRACtest_delete_snapshot (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14672025-04-30T06:42:23.440ZDEBGtest_delete_snapshot (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14682025-04-30T06:42:23.440ZDEBGtest_delete_snapshot (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14692025-04-30T06:42:23.440ZDEBGtest_delete_snapshot (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14702025-04-30T06:42:23.455ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d25c0109-28f4-495a-9f4a-60aa45faae2a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14712025-04-30T06:42:23.455ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
14722025-04-30T06:42:23.456ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1312d5d5-ef03-498c-bfc1-460b05bb8155", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14732025-04-30T06:42:23.456ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
14742025-04-30T06:42:23.458ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd45fcf1-38e1-4aad-83b2-2e0c1f079925", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14752025-04-30T06:42:23.458ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
14762025-04-30T06:42:23.459ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be032c10-09df-4aaa-b388-e1a394e5323a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14772025-04-30T06:42:23.460ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
14782025-04-30T06:42:23.462ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a38c917-5f95-4950-b053-f9a9c663083d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14792025-04-30T06:42:23.462ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
14802025-04-30T06:42:23.475ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3f01e07-5fad-491f-a94a-48bd305ec350", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14812025-04-30T06:42:23.475ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
14822025-04-30T06:42:23.479ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44c068e1-2573-4890-8780-a43dee533a3b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14832025-04-30T06:42:23.479ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
14842025-04-30T06:42:23.482ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4efca75c-0567-45eb-ada4-dc0ba156f075", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14852025-04-30T06:42:23.483ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
14862025-04-30T06:42:23.487ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b4c14ba-9709-44ec-9f4b-5c92a9a2126c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14872025-04-30T06:42:23.487ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
14882025-04-30T06:42:23.493ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7a2dc3a-7b8d-4210-a174-c1a65a74d776", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14892025-04-30T06:42:23.493ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
14902025-04-30T06:42:23.495ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e48a3bb-8cb9-4755-b0a2-0643e547cafb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14912025-04-30T06:42:23.496ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
14922025-04-30T06:42:23.498ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4b2b601-a87a-4238-8bb8-f8a3ac020f9b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14932025-04-30T06:42:23.498ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
14942025-04-30T06:42:23.500ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "136bae6e-7601-48fd-bf3d-91f5e10fe0a6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14952025-04-30T06:42:23.500ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
14962025-04-30T06:42:23.501ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ff0037d-1756-4e61-8b82-910df2af5611", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14972025-04-30T06:42:23.501ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
14982025-04-30T06:42:23.505ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "abc38cd6-c841-433e-b6e8-916e12370f60", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
14992025-04-30T06:42:23.506ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
15002025-04-30T06:42:23.520ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59411a67-aa23-4fc6-8454-82a705b42e99", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15012025-04-30T06:42:23.520ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
15022025-04-30T06:42:23.524ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dbc89a1-7b9e-440e-a795-bf91b449a7f1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15032025-04-30T06:42:23.525ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
15042025-04-30T06:42:23.547ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bbe5f23-c254-43aa-ad4a-edfa8f2c5fd0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15052025-04-30T06:42:23.547ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
15062025-04-30T06:42:23.551ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82f8a2d8-6a49-46a9-94da-19977e13d5e1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15072025-04-30T06:42:23.551ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
15082025-04-30T06:42:23.556ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "123f37a9-07d9-4c53-b622-1fbe2f942bde", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15092025-04-30T06:42:23.556ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
15102025-04-30T06:42:23.582ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d61368-2293-45bb-907f-2c42f66f155c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15112025-04-30T06:42:23.582ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
15122025-04-30T06:42:23.584ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d036c474-05f3-4a3c-8bc6-8eaf743651ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15132025-04-30T06:42:23.584ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
15142025-04-30T06:42:23.598ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ff62ee8-9a8b-4fc8-93e8-b38f52f7256d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15152025-04-30T06:42:23.598ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
15162025-04-30T06:42:23.600ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f25b283a-a93e-4cf0-9228-93716796859b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15172025-04-30T06:42:23.600ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
15182025-04-30T06:42:23.610ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e701340-483e-47bd-9655-c88b3a43c857", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15192025-04-30T06:42:23.610ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
15202025-04-30T06:42:23.612ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1374d260-0491-466f-ae8d-5f6c51bd479d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15212025-04-30T06:42:23.614ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
15222025-04-30T06:42:23.618ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0e219a5-9150-45a8-a818-30c9259d3882", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15232025-04-30T06:42:23.620ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
15242025-04-30T06:42:23.626ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42fca4d2-a858-4a40-91b4-8821fc8c0203", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15252025-04-30T06:42:23.626ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
15262025-04-30T06:42:23.631ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3027d005-1114-429c-8d24-ff5688ff7bb6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15272025-04-30T06:42:23.633ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
15282025-04-30T06:42:23.640ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "095cc94f-dd3a-4b3d-8d42-4d258d1e32fc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15292025-04-30T06:42:23.640ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
15302025-04-30T06:42:23.641ZINFOtest_delete_snapshot (InternalApi): inserted blueprint blueprint_id = 7955e278-b3f1-4d70-b744-51e03fa278f7
15312025-04-30T06:42:23.641ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
15322025-04-30T06:42:23.641ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15332025-04-30T06:42:23.642ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15342025-04-30T06:42:23.642ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
15352025-04-30T06:42:23.643ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15362025-04-30T06:42:23.645ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2155abe-7aec-4338-88d5-f4de63e05c03", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15372025-04-30T06:42:23.645ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
15382025-04-30T06:42:23.645ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15392025-04-30T06:42:23.663ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15402025-04-30T06:42:23.681ZINFOtest_delete_snapshot (InternalApi): No networking records needed for cockroach_db service
15412025-04-30T06:42:23.681ZINFOtest_delete_snapshot (InternalApi): No networking records needed for crucible_pantry service
15422025-04-30T06:42:23.681ZINFOtest_delete_snapshot (InternalApi): No networking records needed for clickhouse service
15432025-04-30T06:42:23.683ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8610b9ce-acea-46fe-8cde-e15777e2293d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15442025-04-30T06:42:23.683ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
15452025-04-30T06:42:23.685ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c425aeb-338c-461b-92c1-9f6b530fb7c2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15462025-04-30T06:42:23.685ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
15472025-04-30T06:42:23.687ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42ea72b4-6569-4de8-81dd-918502e8278f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15482025-04-30T06:42:23.687ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
15492025-04-30T06:42:23.737ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec6c31c1-6b7d-4b24-affd-308e5960f9af", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15502025-04-30T06:42:23.738ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
15512025-04-30T06:42:23.743ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30aaf7a9-0544-4806-a618-64aa82db15fa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15522025-04-30T06:42:23.743ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15532025-04-30T06:42:23.744ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15542025-04-30T06:42:23.744ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15552025-04-30T06:42:23.744ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15562025-04-30T06:42:23.745ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15572025-04-30T06:42:23.745ZDEBGtest_delete_snapshot (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
15582025-04-30T06:42:23.745ZDEBGtest_delete_snapshot (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
15592025-04-30T06:42:23.745ZINFOtest_delete_snapshot (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
15602025-04-30T06:42:23.745ZDEBGtest_delete_snapshot (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 5.883648044s
15612025-04-30T06:42:23.745ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 5.883730208s iteration = 1 status = {}
15622025-04-30T06:42:23.772ZINFOtest_delete_snapshot (InternalApi): Inserted networking records for external_dns service
15632025-04-30T06:42:23.783ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15642025-04-30T06:42:23.789ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15652025-04-30T06:42:23.789ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15662025-04-30T06:42:23.789ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15672025-04-30T06:42:23.789ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15682025-04-30T06:42:23.789ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15692025-04-30T06:42:23.790ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15702025-04-30T06:42:23.801ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15712025-04-30T06:42:23.809ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15722025-04-30T06:42:23.810ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15732025-04-30T06:42:23.810ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15742025-04-30T06:42:23.811ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15752025-04-30T06:42:23.892ZINFOtest_delete_snapshot (InternalApi): Inserted networking records for nexus service
15762025-04-30T06:42:23.892ZINFOtest_delete_snapshot (InternalApi): No networking records needed for internal_dns service
15772025-04-30T06:42:23.892ZINFOtest_delete_snapshot (InternalApi): Inserted service networking records
15782025-04-30T06:42:23.892ZINFOtest_delete_snapshot (InternalApi): Inserted physical disks
15792025-04-30T06:42:23.892ZINFOtest_delete_snapshot (InternalApi): Inserted zpools
15802025-04-30T06:42:23.892ZINFOtest_delete_snapshot (InternalApi): Inserted datasets
15812025-04-30T06:42:23.973ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
15822025-04-30T06:42:23.973ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
15832025-04-30T06:42:23.986ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:62721 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15842025-04-30T06:42:23.987ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:62721 req_id = a56bebfa-98ff-496d-9429-5bc3a4632e83 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15852025-04-30T06:42:23.988ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 695 local_addr = [::1]:62607 method = GET remote_addr = [::1]:62721 req_id = a56bebfa-98ff-496d-9429-5bc3a4632e83 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15862025-04-30T06:42:23.988ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a56bebfa-98ff-496d-9429-5bc3a4632e83", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:42:23 GMT"} })
15872025-04-30T06:42:23.992ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15882025-04-30T06:42:23.992ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15892025-04-30T06:42:23.992ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
15902025-04-30T06:42:23.995ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
15912025-04-30T06:42:23.995ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15922025-04-30T06:42:23.995ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15932025-04-30T06:42:23.995ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15942025-04-30T06:42:23.996ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15952025-04-30T06:42:24.002ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15962025-04-30T06:42:24.007ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
15972025-04-30T06:42:24.007ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15982025-04-30T06:42:24.007ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15992025-04-30T06:42:24.008ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16002025-04-30T06:42:24.058ZINFOtest_delete_snapshot (InternalApi): Populated DNS tables for internal DNS
16012025-04-30T06:42:24.092ZINFOtest_delete_snapshot (InternalApi): Populated DNS tables for external DNS
16022025-04-30T06:42:24.108ZTRACtest_delete_snapshot (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16032025-04-30T06:42:24.108ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16042025-04-30T06:42:24.108ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16052025-04-30T06:42:24.108ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16062025-04-30T06:42:24.111ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16072025-04-30T06:42:24.120ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16082025-04-30T06:42:24.123ZDEBGtest_delete_snapshot (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16092025-04-30T06:42:24.123ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16102025-04-30T06:42:24.123ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16112025-04-30T06:42:24.123ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16122025-04-30T06:42:24.123ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16132025-04-30T06:42:24.123ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16142025-04-30T06:42:24.166ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16152025-04-30T06:42:24.169ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16162025-04-30T06:42:24.224ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16172025-04-30T06:42:24.235ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16182025-04-30T06:42:24.235ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16192025-04-30T06:42:24.235ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16202025-04-30T06:42:24.237ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16212025-04-30T06:42:24.252ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16222025-04-30T06:42:24.259ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16232025-04-30T06:42:24.259ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16242025-04-30T06:42:24.259ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16252025-04-30T06:42:24.259ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16262025-04-30T06:42:24.259ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16272025-04-30T06:42:24.260ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16282025-04-30T06:42:24.265ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16292025-04-30T06:42:24.272ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16302025-04-30T06:42:24.312ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16312025-04-30T06:42:24.312ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322025-04-30T06:42:24.312ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16332025-04-30T06:42:24.312ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16342025-04-30T06:42:24.313ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16352025-04-30T06:42:24.367ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362025-04-30T06:42:24.373ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16372025-04-30T06:42:24.403ZINFOtest_delete_snapshot (InternalApi): Created recovery silo
16382025-04-30T06:42:24.404ZINFOtest_delete_snapshot (InternalApi): Created recovery user
16392025-04-30T06:42:24.410ZINFOtest_delete_snapshot (InternalApi): Created recovery user's password
16402025-04-30T06:42:24.410ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 740f8356-b178-4a61-aa64-265ebc80db30, lookup_type: ById(740f8356-b178-4a61-aa64-265ebc80db30) }
16412025-04-30T06:42:24.410ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 740f8356-b178-4a61-aa64-265ebc80db30 resource_type = Silo
16422025-04-30T06:42:24.410ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:42:24.410ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:42:24.427ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:42:24.432ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:42:24.432ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:42:24.440ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:42:24.441ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:42:24.444ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:42:24.449ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 740f8356-b178-4a61-aa64-265ebc80db30, lookup_type: ById(740f8356-b178-4a61-aa64-265ebc80db30) } result = Ok(())
16512025-04-30T06:42:24.449ZDEBGtest_delete_snapshot (InternalApi): Generated role assignment queries
16522025-04-30T06:42:24.461ZINFOtest_delete_snapshot (InternalApi): Granted Silo privileges
16532025-04-30T06:42:24.461ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:42:24.461ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:42:24.461ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:42:24.461ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:42:24.461ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:42:24.464ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:42:24.467ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:42:24.476ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:42:24.477ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:42:24.477ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:42:24.486ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:42:24.487ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:42:24.487ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:42:24.489ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:42:24.489ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:42:24.490ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:42:24.497ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:42:24.497ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:42:24.497ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:42:24.497ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:42:24.499ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:42:24.502ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:42:24.502ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:42:24.502ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:42:24.503ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:42:24.507ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:42:24.507ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:42:24.507ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:42:24.507ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:42:24.509ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:42:24.517ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:42:24.517ZTRACtest_delete_snapshot (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:42:24.517ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:42:24.517ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:42:24.517ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:42:24.517ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:42:24.518ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:42:24.521ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:42:24.521ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:42:24.521ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:42:24.522ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:42:24.538ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16952025-04-30T06:42:24.538ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16962025-04-30T06:42:24.538ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
16972025-04-30T06:42:24.538ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16982025-04-30T06:42:24.544ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-04-30T06:42:24.552ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17002025-04-30T06:42:24.552ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17012025-04-30T06:42:24.552ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17022025-04-30T06:42:24.553ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17032025-04-30T06:42:24.557ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:42:24.557ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:42:24.558ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:42:24.564ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17072025-04-30T06:42:24.564ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17082025-04-30T06:42:24.565ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17092025-04-30T06:42:24.571ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17102025-04-30T06:42:24.571ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17112025-04-30T06:42:24.571ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17122025-04-30T06:42:24.575ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17132025-04-30T06:42:24.575ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17142025-04-30T06:42:24.575ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:42:24.575ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:42:24.576ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:42:24.579ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17182025-04-30T06:42:24.579ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17192025-04-30T06:42:24.579ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17202025-04-30T06:42:24.580ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17212025-04-30T06:42:24.582ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222025-04-30T06:42:24.582ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:42:24.582ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:42:24.583ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:42:24.586ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-04-30T06:42:24.596ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17272025-04-30T06:42:24.596ZTRACtest_delete_snapshot (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17282025-04-30T06:42:24.596ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17292025-04-30T06:42:24.596ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17302025-04-30T06:42:24.597ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17312025-04-30T06:42:24.599ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17322025-04-30T06:42:24.599ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17332025-04-30T06:42:24.600ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17342025-04-30T06:42:24.604ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17352025-04-30T06:42:24.604ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17362025-04-30T06:42:24.605ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17372025-04-30T06:42:24.607ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17382025-04-30T06:42:24.607ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17392025-04-30T06:42:24.608ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17402025-04-30T06:42:24.611ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17412025-04-30T06:42:24.611ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17422025-04-30T06:42:24.611ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:42:24.611ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:42:24.611ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:42:24.612ZINFOtest_delete_snapshot (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 6e5eb653-feb7-4a7f-81f2-73b2ee16696b
17462025-04-30T06:42:24.612ZDEBGtest_delete_snapshot (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 6e5eb653-feb7-4a7f-81f2-73b2ee16696b time_started = 2025-04-30 06:42:22.790111 UTC
17472025-04-30T06:42:24.613ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 5.755897399s iteration = 2 status = {"collection_id":"6e5eb653-feb7-4a7f-81f2-73b2ee16696b","time_done":"2025-04-30 06:42:23.992685 UTC","time_started":"2025-04-30 06:42:22.790111 UTC"}
17482025-04-30T06:42:24.613ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
17492025-04-30T06:42:24.613ZWARNtest_delete_snapshot (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
17502025-04-30T06:42:24.613ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 29.148\u{b5}s iteration = 2 status = {"error":"no blueprint"}
17512025-04-30T06:42:24.613ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
17522025-04-30T06:42:24.613ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 12.094\u{b5}s iteration = 2 status = {"error":"task disabled"}
17532025-04-30T06:42:24.614ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17542025-04-30T06:42:24.614ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17552025-04-30T06:42:24.614ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17562025-04-30T06:42:24.615ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17572025-04-30T06:42:24.617ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17582025-04-30T06:42:24.617ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17592025-04-30T06:42:24.617ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17602025-04-30T06:42:24.618ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17612025-04-30T06:42:24.620ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17622025-04-30T06:42:24.641ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17632025-04-30T06:42:24.641ZTRACtest_delete_snapshot (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17642025-04-30T06:42:24.641ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17652025-04-30T06:42:24.642ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17662025-04-30T06:42:24.642ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17672025-04-30T06:42:24.642ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17682025-04-30T06:42:24.642ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17692025-04-30T06:42:24.650ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17702025-04-30T06:42:24.650ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17712025-04-30T06:42:24.650ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17722025-04-30T06:42:24.651ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17732025-04-30T06:42:24.655ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17742025-04-30T06:42:24.655ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:42:24.655ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:42:24.656ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:42:24.663ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17782025-04-30T06:42:24.678ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17792025-04-30T06:42:24.678ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17802025-04-30T06:42:24.678ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17812025-04-30T06:42:24.679ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17822025-04-30T06:42:24.695ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17832025-04-30T06:42:24.695ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17842025-04-30T06:42:24.695ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17852025-04-30T06:42:24.712ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17862025-04-30T06:42:24.712ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17872025-04-30T06:42:24.713ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17882025-04-30T06:42:24.719ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17892025-04-30T06:42:24.719ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17902025-04-30T06:42:24.719ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17912025-04-30T06:42:24.736ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17922025-04-30T06:42:24.736ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17932025-04-30T06:42:24.737ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17942025-04-30T06:42:24.742ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17952025-04-30T06:42:24.743ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17962025-04-30T06:42:24.743ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:42:24.743ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:42:24.750ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:42:24.755ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18002025-04-30T06:42:24.756ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18012025-04-30T06:42:24.756ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
18022025-04-30T06:42:24.757ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18032025-04-30T06:42:24.761ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18042025-04-30T06:42:24.761ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:42:24.761ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:42:24.762ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:42:24.764ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18082025-04-30T06:42:24.777ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18092025-04-30T06:42:24.781ZTRACtest_delete_snapshot (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18102025-04-30T06:42:24.781ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
18112025-04-30T06:42:24.781ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18122025-04-30T06:42:24.781ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:42:24.781ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:42:24.782ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:42:24.785ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18162025-04-30T06:42:24.785ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18172025-04-30T06:42:24.786ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
18182025-04-30T06:42:24.786ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18192025-04-30T06:42:24.790ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18202025-04-30T06:42:24.790ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18212025-04-30T06:42:24.790ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
18222025-04-30T06:42:24.791ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18232025-04-30T06:42:24.794ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-04-30T06:42:24.802ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18252025-04-30T06:42:24.802ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18262025-04-30T06:42:24.802ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
18272025-04-30T06:42:24.803ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18282025-04-30T06:42:24.814ZDEBGtest_delete_snapshot (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 7a19b9f1-6cbb-4bcf-bf9e-81f32c6b51a4, kind: Service { id: de3c7c83-8252-43f4-ba2e-cbd92004c4b0 }, name: Name("nexus-de3c7c83-8252-43f4-ba2e-cbd92004c4b0"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 59be950a-353b-4a15-b17c-d5e99051e109, kind: Service { id: ca761d4f-8fbb-4386-a3ef-90926f06ae50 }, name: Name("external-dns-ca761d4f-8fbb-4386-a3ef-90926f06ae50"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
18292025-04-30T06:42:24.814ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18302025-04-30T06:42:24.814ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18312025-04-30T06:42:24.814ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18322025-04-30T06:42:24.814ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
18332025-04-30T06:42:24.815ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18342025-04-30T06:42:24.818ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18352025-04-30T06:42:24.824ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18362025-04-30T06:42:24.824ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18372025-04-30T06:42:24.824ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
18382025-04-30T06:42:24.825ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18392025-04-30T06:42:24.827ZTRACtest_delete_snapshot (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:42:24.467305Z, allowed_ips: None }
18402025-04-30T06:42:24.827ZDEBGtest_delete_snapshot (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:42:24.828ZDEBGtest_delete_snapshot (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 59be950a-353b-4a15-b17c-d5e99051e109, kind: Service { id: ca761d4f-8fbb-4386-a3ef-90926f06ae50 }, name: Name("external-dns-ca761d4f-8fbb-4386-a3ef-90926f06ae50"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 7a19b9f1-6cbb-4bcf-bf9e-81f32c6b51a4, kind: Service { id: de3c7c83-8252-43f4-ba2e-cbd92004c4b0 }, name: Name("nexus-de3c7c83-8252-43f4-ba2e-cbd92004c4b0"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:42:24.828ZDEBGtest_delete_snapshot (nexus): resolved 2 rules for sleds
18432025-04-30T06:42:24.866ZDEBGtest_delete_snapshot (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:42:17.957295Z, time_modified: 2025-04-30T06:42:17.957295Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(62607), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(62141) }]
18442025-04-30T06:42:24.866ZDEBGtest_delete_snapshot (nexus): sending firewall rules to sled agents
18452025-04-30T06:42:24.866ZTRACtest_delete_snapshot (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:42:24.866ZDEBGtest_delete_snapshot (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:42:24.867ZDEBGtest_delete_snapshot (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:42:24.871ZTRACtest_delete_snapshot (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:42:24.871ZTRACtest_delete_snapshot (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:42:24.871ZTRACtest_delete_snapshot (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:42:24.871ZDEBGtest_delete_snapshot (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:42:24.872ZDEBGtest_delete_snapshot (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:42:24.875ZDEBGtest_delete_snapshot (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:42:24.881ZDEBGtest_delete_snapshot (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:42:24.935ZDEBGtest_delete_snapshot (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:62607/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:42:24.936ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:54809 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:42:24.937ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = PUT remote_addr = [::1]:54809 req_id = ea786b4a-2012-49ae-873e-054c9af42fcb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:42:24.939ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1967 local_addr = [::1]:62607 method = PUT remote_addr = [::1]:54809 req_id = ea786b4a-2012-49ae-873e-054c9af42fcb response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:42:24.939ZDEBGtest_delete_snapshot (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:62607/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "ea786b4a-2012-49ae-873e-054c9af42fcb", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
18602025-04-30T06:42:24.941ZINFOtest_delete_snapshot (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:42:24.941ZTRACtest_delete_snapshot (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:42:24.941ZDEBGtest_delete_snapshot (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:42:24.942ZDEBGtest_delete_snapshot (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:42:24.942ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:42:24.942ZDEBGtest_delete_snapshot (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:42:24.942ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:42:24.942ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:42:24.943ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:42:24.943ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:42:24.943ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:42:24.943ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:42:24.943ZDEBGtest_delete_snapshot (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:42:24.943ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:42:24.943ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:42:24.944ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:42:24.944ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:42:24.944ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:42:24.944ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:42:24.944ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:42:24.944ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:42:24.944ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:42:24.944ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:42:24.944ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:42:24.945ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:42:24.945ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:42:24.945ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:42:24.945ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:42:24.945ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:42:24.945ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:42:24.947ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 07 31 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 .1..........._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:58031 req_id = 674258ef-d23f-4362-8c28-69e7cad5e4b0
18912025-04-30T06:42:24.947ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1841,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58031 req_id = 674258ef-d23f-4362-8c28-69e7cad5e4b0
18922025-04-30T06:42:24.947ZDEBGtest_delete_snapshot (store): query key key = _nameservice._tcp
18932025-04-30T06:42:24.947ZDEBGtest_delete_snapshot (store): query key key = f08867ef-19ac-4764-90de-9146c69b8f3f.host
18942025-04-30T06:42:24.947ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58031 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 51007, target: Name("f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal") })) }] req_id = 674258ef-d23f-4362-8c28-69e7cad5e4b0
18952025-04-30T06:42:24.948ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 8b eb 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50042 req_id = fd031d98-b5d0-42a4-97b9-fa43e80a01f5
18962025-04-30T06:42:24.948ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35819,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50042 req_id = fd031d98-b5d0-42a4-97b9-fa43e80a01f5
18972025-04-30T06:42:24.948ZDEBGtest_delete_snapshot (store): query key key = _external-dns._tcp
18982025-04-30T06:42:24.949ZDEBGtest_delete_snapshot (store): query key key = ca761d4f-8fbb-4386-a3ef-90926f06ae50.host
18992025-04-30T06:42:24.949ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50042 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41697, target: Name("ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal") })) }] req_id = fd031d98-b5d0-42a4-97b9-fa43e80a01f5
19002025-04-30T06:42:24.949ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19012025-04-30T06:42:24.949ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19022025-04-30T06:42:24.949ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19032025-04-30T06:42:24.949ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19042025-04-30T06:42:24.950ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19052025-04-30T06:42:24.950ZTRACtest_delete_snapshot (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19062025-04-30T06:42:24.950ZTRACtest_delete_snapshot (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19072025-04-30T06:42:24.950ZTRACtest_delete_snapshot (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19082025-04-30T06:42:24.951ZDEBGtest_delete_snapshot (ServiceBalancer): roles roles = RoleSet { roles: {} }
19092025-04-30T06:42:24.951ZDEBGtest_delete_snapshot (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19102025-04-30T06:42:24.951ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 51007, target: Name("f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4209, tv_nsec: 750742446 } })
19112025-04-30T06:42:24.951ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
19122025-04-30T06:42:24.952ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41697, target: Name("ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4209, tv_nsec: 751107233 } })
19132025-04-30T06:42:24.952ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
19142025-04-30T06:42:24.952ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19152025-04-30T06:42:24.952ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19162025-04-30T06:42:24.952ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19172025-04-30T06:42:24.952ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19182025-04-30T06:42:24.953ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19192025-04-30T06:42:24.953ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 3d 75 01 00 00 01 00 00 00 00 00 01 24 66 30 38 =u..........$f08\n0010: 38 36 37 65 66 2d 31 39 61 63 2d 34 37 36 34 2d 867ef-19ac-4764-\n0020: 39 30 64 65 2d 39 31 34 36 63 36 39 62 38 66 33 90de-9146c69b8f3\n0030: 66 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 f.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:64456 req_id = fc04717a-e463-4c34-984b-73c9bca6e918
19202025-04-30T06:42:24.953ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15733,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 102,\n 48,\n 56,\n 56,\n 54,\n 55,\n 101,\n 102,\n 45,\n 49,\n 57,\n 97,\n 99,\n 45,\n 52,\n 55,\n 54,\n 52,\n 45,\n 57,\n 48,\n 100,\n 101,\n 45,\n 57,\n 49,\n 52,\n 54,\n 99,\n 54,\n 57,\n 98,\n 56,\n 102,\n 51,\n 102,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64456 req_id = fc04717a-e463-4c34-984b-73c9bca6e918
19212025-04-30T06:42:24.953ZDEBGtest_delete_snapshot (store): query key key = f08867ef-19ac-4764-90de-9146c69b8f3f.host
19222025-04-30T06:42:24.954ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64456 query = LowerQuery { name: LowerName(Name("f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal.")), original: Query { name: Name("f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("f08867ef-19ac-4764-90de-9146c69b8f3f.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fc04717a-e463-4c34-984b-73c9bca6e918
19232025-04-30T06:42:24.954ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 2a 74 01 00 00 01 00 00 00 00 00 01 24 63 61 37 *t..........$ca7\n0010: 36 31 64 34 66 2d 38 66 62 62 2d 34 33 38 36 2d 61d4f-8fbb-4386-\n0020: 61 33 65 66 2d 39 30 39 32 36 66 30 36 61 65 35 a3ef-90926f06ae5\n0030: 30 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 0.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:55742 req_id = dde0555e-8728-4230-a6a0-43644b51515f
19242025-04-30T06:42:24.954ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10868,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 99,\n 97,\n 55,\n 54,\n 49,\n 100,\n 52,\n 102,\n 45,\n 56,\n 102,\n 98,\n 98,\n 45,\n 52,\n 51,\n 56,\n 54,\n 45,\n 97,\n 51,\n 101,\n 102,\n 45,\n 57,\n 48,\n 57,\n 50,\n 54,\n 102,\n 48,\n 54,\n 97,\n 101,\n 53,\n 48,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55742 req_id = dde0555e-8728-4230-a6a0-43644b51515f
19252025-04-30T06:42:24.954ZDEBGtest_delete_snapshot (store): query key key = ca761d4f-8fbb-4386-a3ef-90926f06ae50.host
19262025-04-30T06:42:24.954ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55742 query = LowerQuery { name: LowerName(Name("ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal.")), original: Query { name: Name("ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("ca761d4f-8fbb-4386-a3ef-90926f06ae50.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dde0555e-8728-4230-a6a0-43644b51515f
19272025-04-30T06:42:24.956ZINFOtest_delete_snapshot (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:51007] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:42:24.956ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 12.591057ms iteration = 2 status = {"addresses":["[::1]:51007"]}
19292025-04-30T06:42:24.956ZINFOtest_delete_snapshot (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:41697] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:42:24.956ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 12.093566ms iteration = 2 status = {"addresses":["[::1]:41697"]}
19312025-04-30T06:42:24.956ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:42:24.956ZWARNtest_delete_snapshot (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:42:24.956ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 24.008\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:42:24.956ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:42:24.956ZWARNtest_delete_snapshot (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:42:24.956ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 11.493\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:42:24.956ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:42:24.965ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19392025-04-30T06:42:24.965ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19402025-04-30T06:42:24.965ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19412025-04-30T06:42:24.965ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19422025-04-30T06:42:24.965ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:42:24.970ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19442025-04-30T06:42:24.970ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19452025-04-30T06:42:24.970ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19462025-04-30T06:42:24.971ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19472025-04-30T06:42:24.974ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19482025-04-30T06:42:24.983ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19492025-04-30T06:42:24.985ZDEBGtest_delete_snapshot (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19502025-04-30T06:42:24.985ZDEBGtest_delete_snapshot (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19512025-04-30T06:42:24.985ZDEBGtest_delete_snapshot (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19522025-04-30T06:42:24.985ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19532025-04-30T06:42:24.985ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19542025-04-30T06:42:24.985ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19552025-04-30T06:42:24.985ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19562025-04-30T06:42:24.985ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19572025-04-30T06:42:24.996ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19582025-04-30T06:42:25.004ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19592025-04-30T06:42:25.008ZDEBGtest_delete_snapshot (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19602025-04-30T06:42:25.015ZDEBGtest_delete_snapshot (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
19612025-04-30T06:42:25.015ZINFOtest_delete_snapshot (nexus): Rack initialized
19622025-04-30T06:42:25.015ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19632025-04-30T06:42:25.015ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
19642025-04-30T06:42:25.016ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19652025-04-30T06:42:25.016ZDEBGtest_delete_snapshot (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19662025-04-30T06:42:25.016ZDEBGtest_delete_snapshot (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19672025-04-30T06:42:25.017ZDEBGtest_delete_snapshot (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19682025-04-30T06:42:25.017ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19692025-04-30T06:42:25.017ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19702025-04-30T06:42:25.017ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19712025-04-30T06:42:25.017ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19722025-04-30T06:42:25.018ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19732025-04-30T06:42:25.018ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
19742025-04-30T06:42:25.018ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19752025-04-30T06:42:25.018ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19762025-04-30T06:42:25.018ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19772025-04-30T06:42:25.019ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19782025-04-30T06:42:25.020ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
19792025-04-30T06:42:25.020ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19802025-04-30T06:42:25.021ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19812025-04-30T06:42:25.027ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19822025-04-30T06:42:25.027ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19832025-04-30T06:42:25.027ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19842025-04-30T06:42:25.034ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19852025-04-30T06:42:25.037ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19862025-04-30T06:42:25.044ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19872025-04-30T06:42:25.044ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19882025-04-30T06:42:25.044ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19892025-04-30T06:42:25.045ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19902025-04-30T06:42:25.046ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19912025-04-30T06:42:25.051ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19922025-04-30T06:42:25.051ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19932025-04-30T06:42:25.051ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19942025-04-30T06:42:25.051ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19952025-04-30T06:42:25.056ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 50 e3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 P............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54116 req_id = 53abca5e-1363-4678-914c-1e37f91abf4e
19962025-04-30T06:42:25.056ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20707,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54116 req_id = 53abca5e-1363-4678-914c-1e37f91abf4e
19972025-04-30T06:42:25.056ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
19982025-04-30T06:42:25.056ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19992025-04-30T06:42:25.056ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54116 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 53abca5e-1363-4678-914c-1e37f91abf4e
20002025-04-30T06:42:25.056ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4209, tv_nsec: 853446467 } })
20012025-04-30T06:42:25.056ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
20022025-04-30T06:42:25.056ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ad f9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57295 req_id = 2fa50c8f-bd97-4e14-b527-33dee26afa7c
20032025-04-30T06:42:25.056ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44537,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57295 req_id = 2fa50c8f-bd97-4e14-b527-33dee26afa7c
20042025-04-30T06:42:25.056ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20052025-04-30T06:42:25.056ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57295 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2fa50c8f-bd97-4e14-b527-33dee26afa7c
20062025-04-30T06:42:25.138ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
20072025-04-30T06:42:25.139ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20082025-04-30T06:42:25.139ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20092025-04-30T06:42:25.139ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20102025-04-30T06:42:25.139ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20112025-04-30T06:42:25.140ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20122025-04-30T06:42:25.141ZDEBGtest_delete_snapshot (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
20132025-04-30T06:42:25.141ZDEBGtest_delete_snapshot (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fc6af4ff-999f-4008-98bf-a7d9b5874136 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
20142025-04-30T06:42:25.141ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
20152025-04-30T06:42:25.141ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20162025-04-30T06:42:25.141ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20172025-04-30T06:42:25.141ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20182025-04-30T06:42:25.142ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20192025-04-30T06:42:25.142ZDEBGtest_delete_snapshot (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20202025-04-30T06:42:25.142ZDEBGtest_delete_snapshot (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = fa46fc9c-e76a-49c5-97c2-9222a929d181 dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
20212025-04-30T06:42:25.142ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
20222025-04-30T06:42:25.142ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20232025-04-30T06:42:25.142ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20242025-04-30T06:42:25.143ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20252025-04-30T06:42:25.143ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20262025-04-30T06:42:25.144ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20272025-04-30T06:42:25.144ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
20282025-04-30T06:42:25.144ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20292025-04-30T06:42:25.147ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 51 06 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Q............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58557 req_id = 1a29f1c0-8c77-462d-8246-33a853077548
20302025-04-30T06:42:25.147ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20742,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58557 req_id = 1a29f1c0-8c77-462d-8246-33a853077548
20312025-04-30T06:42:25.147ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
20322025-04-30T06:42:25.147ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20742, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58557 req_id = 1a29f1c0-8c77-462d-8246-33a853077548
20332025-04-30T06:42:25.148ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
20342025-04-30T06:42:25.148ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
20352025-04-30T06:42:25.148ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
20362025-04-30T06:42:25.170ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20372025-04-30T06:42:25.175ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20382025-04-30T06:42:25.175ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20392025-04-30T06:42:25.175ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20402025-04-30T06:42:25.176ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20412025-04-30T06:42:25.176ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20422025-04-30T06:42:25.182ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20432025-04-30T06:42:25.182ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20442025-04-30T06:42:25.182ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20452025-04-30T06:42:25.183ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20462025-04-30T06:42:25.183ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20472025-04-30T06:42:25.188ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20482025-04-30T06:42:25.188ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20492025-04-30T06:42:25.188ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20502025-04-30T06:42:25.189ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20512025-04-30T06:42:25.190ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "68db3e3e-0a66-4b58-94ad-5e84ea8964aa", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
20522025-04-30T06:42:25.191ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
20532025-04-30T06:42:25.201ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20542025-04-30T06:42:25.201ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
20552025-04-30T06:42:25.201ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20562025-04-30T06:42:25.224ZDEBGtest_delete_snapshot (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fc6af4ff-999f-4008-98bf-a7d9b5874136 dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20572025-04-30T06:42:25.224ZDEBGtest_delete_snapshot (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20582025-04-30T06:42:25.224ZINFOtest_delete_snapshot (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20592025-04-30T06:42:25.224ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 281.572134ms iteration = 2 status = {"generation":1}
20602025-04-30T06:42:25.224ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20612025-04-30T06:42:25.270ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:51007 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:51007] } uri = http://[::1]:51007/config
20622025-04-30T06:42:25.271ZDEBGtest_delete_snapshot (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = fa46fc9c-e76a-49c5-97c2-9222a929d181 dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20632025-04-30T06:42:25.271ZDEBGtest_delete_snapshot (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20642025-04-30T06:42:25.271ZINFOtest_delete_snapshot (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20652025-04-30T06:42:25.271ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 327.825471ms iteration = 2 status = {"generation":2}
20662025-04-30T06:42:25.271ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20672025-04-30T06:42:25.331ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:41697 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:41697] } uri = http://[::1]:41697/config
20682025-04-30T06:42:25.333ZINFOtest_delete_snapshot (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:42:16.485776Z, time_modified: 2025-04-30T06:42:16.485776Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 740f8356-b178-4a61-aa64-265ebc80db30, db_silo: Silo { identity: SiloIdentity { id: 740f8356-b178-4a61-aa64-265ebc80db30, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:42:24.107966Z, time_modified: 2025-04-30T06:42:24.107966Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: 740f8356-b178-4a61-aa64-265ebc80db30, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 740f8356-b178-4a61-aa64-265ebc80db30, db_silo: Silo { identity: SiloIdentity { id: 740f8356-b178-4a61-aa64-265ebc80db30, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:42:24.107966Z, time_modified: 2025-04-30T06:42:24.107966Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
20692025-04-30T06:42:25.333ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 389.071951ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"740f8356-b178-4a61-aa64-265ebc80db30","tls_certs":[]}},"default_endpoint":{"silo_id":"740f8356-b178-4a61-aa64-265ebc80db30","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo 740f8356-b178-4a61-aa64-265ebc80db30 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
20702025-04-30T06:42:25.334ZINFOtest_delete_snapshot (http): accepted connection kind = dns local_addr = [::1]:51007 remote_addr = [::1]:43539
20712025-04-30T06:42:25.336ZINFOtest_delete_snapshot (http): accepted connection kind = dns local_addr = [::1]:41697 remote_addr = [::1]:43679
20722025-04-30T06:42:25.343ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20732025-04-30T06:42:25.343ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20742025-04-30T06:42:25.343ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20752025-04-30T06:42:25.343ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
20762025-04-30T06:42:25.344ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20772025-04-30T06:42:25.345ZTRACtest_delete_snapshot (http): incoming request kind = dns local_addr = [::1]:51007 method = PUT remote_addr = [::1]:43539 req_id = d0a8690e-4ebc-46d5-bbef-ba6a284f729f uri = /config
20782025-04-30T06:42:25.347ZTRACtest_delete_snapshot (http): incoming request kind = dns local_addr = [::1]:41697 method = PUT remote_addr = [::1]:43679 req_id = e6b72e04-d9d7-4421-8d8d-e8091683d7c5 uri = /config
20792025-04-30T06:42:25.347ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3926cbc2-bf5d-4d15-8f05-232386104cae", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
20802025-04-30T06:42:25.348ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
20812025-04-30T06:42:25.348ZINFOtest_delete_snapshot (store): attempting generation update new_generation = 1 req_id = d0a8690e-4ebc-46d5-bbef-ba6a284f729f
20822025-04-30T06:42:25.348ZINFOtest_delete_snapshot (store): updated generation new_generation = 1 req_id = d0a8690e-4ebc-46d5-bbef-ba6a284f729f
20832025-04-30T06:42:25.348ZINFOtest_delete_snapshot (store): attempting generation update new_generation = 2 req_id = e6b72e04-d9d7-4421-8d8d-e8091683d7c5
20842025-04-30T06:42:25.348ZINFOtest_delete_snapshot (store): pruning trees for generations newer than 0
20852025-04-30T06:42:25.348ZDEBGtest_delete_snapshot (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20862025-04-30T06:42:25.349ZDEBGtest_delete_snapshot (store): updating current config
20872025-04-30T06:42:25.349ZDEBGtest_delete_snapshot (store): flushing default tree
20882025-04-30T06:42:25.349ZINFOtest_delete_snapshot (store): pruning trees for generations older than 2 keep = 3
20892025-04-30T06:42:25.349ZINFOtest_delete_snapshot (store): updated generation new_generation = 2 req_id = e6b72e04-d9d7-4421-8d8d-e8091683d7c5
20902025-04-30T06:42:25.350ZINFOtest_delete_snapshot (http): request completed kind = dns latency_us = 4454 local_addr = [::1]:51007 method = PUT remote_addr = [::1]:43539 req_id = d0a8690e-4ebc-46d5-bbef-ba6a284f729f response_code = 204 uri = /config
20912025-04-30T06:42:25.350ZINFOtest_delete_snapshot (http): request completed kind = dns latency_us = 2769 local_addr = [::1]:41697 method = PUT remote_addr = [::1]:43679 req_id = e6b72e04-d9d7-4421-8d8d-e8091683d7c5 response_code = 204 uri = /config
20922025-04-30T06:42:25.353ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:51007 generation = 1 result = Ok(Response { url: "http://[::1]:51007/config", status: 204, headers: {"x-request-id": "d0a8690e-4ebc-46d5-bbef-ba6a284f729f", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} }) servers = DnsServersList { addresses: [[::1]:51007] }
20932025-04-30T06:42:25.353ZINFOtest_delete_snapshot (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:51007 generation = 1 servers = DnsServersList { addresses: [[::1]:51007] }
20942025-04-30T06:42:25.354ZINFOtest_delete_snapshot (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:51007] }
20952025-04-30T06:42:25.354ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 130.373922ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:51007":{"Ok":null}}}
20962025-04-30T06:42:25.354ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:41697 generation = 2 result = Ok(Response { url: "http://[::1]:41697/config", status: 204, headers: {"x-request-id": "e6b72e04-d9d7-4421-8d8d-e8091683d7c5", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} }) servers = DnsServersList { addresses: [[::1]:41697] }
20972025-04-30T06:42:25.354ZINFOtest_delete_snapshot (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:41697 generation = 2 servers = DnsServersList { addresses: [[::1]:41697] }
20982025-04-30T06:42:25.356ZINFOtest_delete_snapshot (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:41697] }
20992025-04-30T06:42:25.356ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 85.326016ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:41697":{"Ok":null}}}
21002025-04-30T06:42:25.361ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21012025-04-30T06:42:25.361ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21022025-04-30T06:42:25.361ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21032025-04-30T06:42:25.362ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21042025-04-30T06:42:25.365ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21052025-04-30T06:42:25.365ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21062025-04-30T06:42:25.365ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21072025-04-30T06:42:25.366ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21082025-04-30T06:42:25.368ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c053712d-deb0-489f-89ab-c4d7cc21390a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21092025-04-30T06:42:25.368ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
21102025-04-30T06:42:25.369ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21112025-04-30T06:42:25.379ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21122025-04-30T06:42:25.379ZTRACtest_delete_snapshot (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21132025-04-30T06:42:25.379ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21142025-04-30T06:42:25.379ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21152025-04-30T06:42:25.379ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21162025-04-30T06:42:25.379ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21172025-04-30T06:42:25.379ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21182025-04-30T06:42:25.382ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21192025-04-30T06:42:25.382ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21202025-04-30T06:42:25.382ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21212025-04-30T06:42:25.382ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21222025-04-30T06:42:25.384ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19d86a2f-eff2-4af0-bfd4-70adc1ac7e11", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21232025-04-30T06:42:25.384ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
21242025-04-30T06:42:25.385ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21252025-04-30T06:42:25.385ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21262025-04-30T06:42:25.385ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21272025-04-30T06:42:25.385ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21282025-04-30T06:42:25.388ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21292025-04-30T06:42:25.402ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21302025-04-30T06:42:25.402ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21312025-04-30T06:42:25.403ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21322025-04-30T06:42:25.403ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21332025-04-30T06:42:25.410ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21342025-04-30T06:42:25.410ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21352025-04-30T06:42:25.411ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21362025-04-30T06:42:25.411ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be2aae90-9900-4c52-b650-b3ae08f637f9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21372025-04-30T06:42:25.411ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
21382025-04-30T06:42:25.415ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21392025-04-30T06:42:25.415ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21402025-04-30T06:42:25.416ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21412025-04-30T06:42:25.418ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dfff4ee-8ede-4722-bae2-2c671521c732", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21422025-04-30T06:42:25.418ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
21432025-04-30T06:42:25.419ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21442025-04-30T06:42:25.419ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21452025-04-30T06:42:25.420ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21462025-04-30T06:42:25.423ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8179ab6-f24d-4d2b-8e8e-5a161da51c02", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21472025-04-30T06:42:25.424ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
21482025-04-30T06:42:25.424ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21492025-04-30T06:42:25.424ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21502025-04-30T06:42:25.424ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21512025-04-30T06:42:25.424ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21522025-04-30T06:42:25.425ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21532025-04-30T06:42:25.431ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21542025-04-30T06:42:25.432ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21552025-04-30T06:42:25.432ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21562025-04-30T06:42:25.432ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21572025-04-30T06:42:25.436ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a5c1e5c-3e90-4e5d-9ee5-2d492e88143a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21582025-04-30T06:42:25.437ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
21592025-04-30T06:42:25.440ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21602025-04-30T06:42:25.440ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21612025-04-30T06:42:25.440ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21622025-04-30T06:42:25.441ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21632025-04-30T06:42:25.444ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21642025-04-30T06:42:25.451ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21652025-04-30T06:42:25.452ZTRACtest_delete_snapshot (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21662025-04-30T06:42:25.452ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21672025-04-30T06:42:25.452ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21682025-04-30T06:42:25.453ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21692025-04-30T06:42:25.454ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "392c953a-9232-463f-adcf-a3660cfa5843", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21702025-04-30T06:42:25.454ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
21712025-04-30T06:42:25.456ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90ffd31c-3f01-4899-8122-c4f805474113", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21722025-04-30T06:42:25.457ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
21732025-04-30T06:42:25.457ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21742025-04-30T06:42:25.457ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21752025-04-30T06:42:25.457ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21762025-04-30T06:42:25.459ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6400a43-e9b6-4de2-ba3b-c11ec68b2e95", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21772025-04-30T06:42:25.459ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
21782025-04-30T06:42:25.462ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9421de5-f29c-4a00-b157-f38cd689802b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21792025-04-30T06:42:25.463ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
21802025-04-30T06:42:25.463ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21812025-04-30T06:42:25.464ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21822025-04-30T06:42:25.466ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21832025-04-30T06:42:25.468ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21842025-04-30T06:42:25.468ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21852025-04-30T06:42:25.469ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21862025-04-30T06:42:25.475ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "116fea1c-51d7-49ce-a3de-d2bc8961c04d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21872025-04-30T06:42:25.475ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
21882025-04-30T06:42:25.476ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
21892025-04-30T06:42:25.476ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21902025-04-30T06:42:25.477ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21912025-04-30T06:42:25.477ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21922025-04-30T06:42:25.478ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21932025-04-30T06:42:25.481ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21942025-04-30T06:42:25.481ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21952025-04-30T06:42:25.481ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
21962025-04-30T06:42:25.482ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21972025-04-30T06:42:25.482ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50ce8eda-f39b-4d75-bbd1-85a2462bc510", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
21982025-04-30T06:42:25.483ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
21992025-04-30T06:42:25.485ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22002025-04-30T06:42:25.485ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22012025-04-30T06:42:25.485ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22022025-04-30T06:42:25.486ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22032025-04-30T06:42:25.487ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78676972-898e-4984-9e56-8c2628922bd5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22042025-04-30T06:42:25.487ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
22052025-04-30T06:42:25.488ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22062025-04-30T06:42:25.503ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22072025-04-30T06:42:25.503ZTRACtest_delete_snapshot (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22082025-04-30T06:42:25.503ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22092025-04-30T06:42:25.503ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22102025-04-30T06:42:25.503ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22112025-04-30T06:42:25.503ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22122025-04-30T06:42:25.506ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22132025-04-30T06:42:25.510ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22142025-04-30T06:42:25.510ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22152025-04-30T06:42:25.510ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22162025-04-30T06:42:25.512ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22172025-04-30T06:42:25.513ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22182025-04-30T06:42:25.513ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22192025-04-30T06:42:25.513ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22202025-04-30T06:42:25.514ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22212025-04-30T06:42:25.516ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65523522-3e82-4ad6-b46d-2c9141f8eed7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22222025-04-30T06:42:25.516ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
22232025-04-30T06:42:25.516ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22242025-04-30T06:42:25.525ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22252025-04-30T06:42:25.526ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22262025-04-30T06:42:25.526ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22272025-04-30T06:42:25.526ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22282025-04-30T06:42:25.529ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1199f055-2d65-4b6e-ab01-7e4d2985c6f4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22292025-04-30T06:42:25.529ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
22302025-04-30T06:42:25.531ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22312025-04-30T06:42:25.532ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22322025-04-30T06:42:25.532ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22332025-04-30T06:42:25.537ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22342025-04-30T06:42:25.537ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22352025-04-30T06:42:25.537ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22362025-04-30T06:42:25.538ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50e6eb30-9803-4dab-8586-83e32af5c46e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22372025-04-30T06:42:25.538ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
22382025-04-30T06:42:25.542ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22392025-04-30T06:42:25.543ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22402025-04-30T06:42:25.545ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22412025-04-30T06:42:25.549ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d836c4b-9ffc-4ba8-9593-73fc7482121c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22422025-04-30T06:42:25.551ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
22432025-04-30T06:42:25.551ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22442025-04-30T06:42:25.551ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22452025-04-30T06:42:25.554ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22462025-04-30T06:42:25.557ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "558e7771-9ba8-48cd-a07c-eef9ae7fe6f9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22472025-04-30T06:42:25.557ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
22482025-04-30T06:42:25.558ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22492025-04-30T06:42:25.558ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22502025-04-30T06:42:25.558ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22512025-04-30T06:42:25.558ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22522025-04-30T06:42:25.559ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22532025-04-30T06:42:25.561ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70b576bc-3863-4ddf-81e4-0204dc26aa13", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22542025-04-30T06:42:25.561ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
22552025-04-30T06:42:25.562ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22562025-04-30T06:42:25.563ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22572025-04-30T06:42:25.563ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22582025-04-30T06:42:25.566ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22592025-04-30T06:42:25.569ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4027699e-7d5a-45d1-b255-f20373b5071b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22602025-04-30T06:42:25.569ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
22612025-04-30T06:42:25.570ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22622025-04-30T06:42:25.570ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22632025-04-30T06:42:25.570ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22642025-04-30T06:42:25.571ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22652025-04-30T06:42:25.573ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb0f0643-12fb-499b-b6e6-6a61b9d86c54", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22662025-04-30T06:42:25.574ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
22672025-04-30T06:42:25.574ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22682025-04-30T06:42:25.582ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22692025-04-30T06:42:25.582ZTRACtest_delete_snapshot (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22702025-04-30T06:42:25.582ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22712025-04-30T06:42:25.582ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22722025-04-30T06:42:25.582ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22732025-04-30T06:42:25.582ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22742025-04-30T06:42:25.582ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22752025-04-30T06:42:25.586ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22762025-04-30T06:42:25.586ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22772025-04-30T06:42:25.586ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22782025-04-30T06:42:25.586ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22792025-04-30T06:42:25.586ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d110cc32-d5d9-4a3a-86b9-5966e1fd7938", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22802025-04-30T06:42:25.587ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
22812025-04-30T06:42:25.589ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22822025-04-30T06:42:25.589ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22832025-04-30T06:42:25.589ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22842025-04-30T06:42:25.591ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22852025-04-30T06:42:25.593ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a39e91c8-b109-4e8f-9c3b-b76d79164087", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22862025-04-30T06:42:25.593ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
22872025-04-30T06:42:25.594ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22882025-04-30T06:42:25.607ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22892025-04-30T06:42:25.607ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22902025-04-30T06:42:25.607ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
22912025-04-30T06:42:25.611ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22922025-04-30T06:42:25.615ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab29ddc6-f616-49d0-95fd-9a51c8da019c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22932025-04-30T06:42:25.615ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
22942025-04-30T06:42:25.618ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "181a5103-103f-41ef-af43-18c386d50b54", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
22952025-04-30T06:42:25.619ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
22962025-04-30T06:42:25.619ZDEBGtest_delete_snapshot (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 7a19b9f1-6cbb-4bcf-bf9e-81f32c6b51a4, kind: Service { id: de3c7c83-8252-43f4-ba2e-cbd92004c4b0 }, name: Name("nexus-de3c7c83-8252-43f4-ba2e-cbd92004c4b0"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 59be950a-353b-4a15-b17c-d5e99051e109, kind: Service { id: ca761d4f-8fbb-4386-a3ef-90926f06ae50 }, name: Name("external-dns-ca761d4f-8fbb-4386-a3ef-90926f06ae50"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
22972025-04-30T06:42:25.619ZTRACtest_delete_snapshot (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
22982025-04-30T06:42:25.620ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22992025-04-30T06:42:25.620ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23002025-04-30T06:42:25.620ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
23012025-04-30T06:42:25.620ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23022025-04-30T06:42:25.622ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c63b3690-50e4-4762-b117-84bc09423597", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23032025-04-30T06:42:25.622ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
23042025-04-30T06:42:25.622ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23052025-04-30T06:42:25.626ZDEBGtest_delete_snapshot (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23062025-04-30T06:42:25.626ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23072025-04-30T06:42:25.626ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
23082025-04-30T06:42:25.627ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23092025-04-30T06:42:25.629ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cbb63cb-2997-4809-ac8f-8ca39dd8400e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23102025-04-30T06:42:25.629ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
23112025-04-30T06:42:25.629ZTRACtest_delete_snapshot (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:42:24.467305Z, allowed_ips: None }
23122025-04-30T06:42:25.629ZDEBGtest_delete_snapshot (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23132025-04-30T06:42:25.630ZDEBGtest_delete_snapshot (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 59be950a-353b-4a15-b17c-d5e99051e109, kind: Service { id: ca761d4f-8fbb-4386-a3ef-90926f06ae50 }, name: Name("external-dns-ca761d4f-8fbb-4386-a3ef-90926f06ae50"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 7a19b9f1-6cbb-4bcf-bf9e-81f32c6b51a4, kind: Service { id: de3c7c83-8252-43f4-ba2e-cbd92004c4b0 }, name: Name("nexus-de3c7c83-8252-43f4-ba2e-cbd92004c4b0"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23142025-04-30T06:42:25.630ZDEBGtest_delete_snapshot (InternalApi): resolved 2 rules for sleds
23152025-04-30T06:42:25.659ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6fe9187-7038-4b85-9be7-7563276494b0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23162025-04-30T06:42:25.659ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
23172025-04-30T06:42:25.663ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b0d310c-414f-4337-b6d1-cac2ae301b39", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23182025-04-30T06:42:25.663ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
23192025-04-30T06:42:25.665ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0725ed0-8820-4780-9f61-2e4082addead", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23202025-04-30T06:42:25.665ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
23212025-04-30T06:42:25.670ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e603fb99-8b2a-475c-8b31-a3bfc201224a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23222025-04-30T06:42:25.678ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
23232025-04-30T06:42:25.682ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "947e14f4-f8bf-431c-ace1-6b573def51e4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23242025-04-30T06:42:25.690ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
23252025-04-30T06:42:25.690ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "986198a2-30b5-47b8-bc45-0d7241ef337d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23262025-04-30T06:42:25.690ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
23272025-04-30T06:42:25.690ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3947cb80-1e52-451c-ba2e-01ed879fb1eb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23282025-04-30T06:42:25.690ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
23292025-04-30T06:42:25.692ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba573dc0-6939-4d9b-ac88-6b4335e7586b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23302025-04-30T06:42:25.693ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
23312025-04-30T06:42:25.698ZDEBGtest_delete_snapshot (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:42:17.957295Z, time_modified: 2025-04-30T06:42:17.957295Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(62607), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(62141) }]
23322025-04-30T06:42:25.698ZDEBGtest_delete_snapshot (InternalApi): sending firewall rules to sled agents
23332025-04-30T06:42:25.698ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23342025-04-30T06:42:25.698ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
23352025-04-30T06:42:25.699ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23362025-04-30T06:42:25.702ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4875896-ab0e-4872-9418-00321c62729e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23372025-04-30T06:42:25.703ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
23382025-04-30T06:42:25.704ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23392025-04-30T06:42:25.704ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23402025-04-30T06:42:25.704ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23412025-04-30T06:42:25.704ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
23422025-04-30T06:42:25.712ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23432025-04-30T06:42:25.716ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23442025-04-30T06:42:25.723ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23452025-04-30T06:42:25.789ZDEBGtest_delete_snapshot (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:62607/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23462025-04-30T06:42:25.790ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:37575 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23472025-04-30T06:42:25.792ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = PUT remote_addr = [::1]:37575 req_id = 2f867f1e-6ca8-4126-be14-6fa7976dbdc2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23482025-04-30T06:42:25.793ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1048 local_addr = [::1]:62607 method = PUT remote_addr = [::1]:37575 req_id = 2f867f1e-6ca8-4126-be14-6fa7976dbdc2 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23492025-04-30T06:42:25.793ZDEBGtest_delete_snapshot (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:62607/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "2f867f1e-6ca8-4126-be14-6fa7976dbdc2", "date": "Wed, 30 Apr 2025 06:42:24 GMT"} })
23502025-04-30T06:42:25.795ZINFOtest_delete_snapshot (InternalApi): updated firewall rules on 1 sleds
23512025-04-30T06:42:25.795ZINFOtest_delete_snapshot (nexus): plumbed initial IP allowlist
23522025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = / versions = all
23532025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /access versions = all
23542025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /assets/{path} versions = all
23552025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /device/auth versions = all
23562025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /device/confirm versions = all
23572025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /device/success versions = all
23582025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /device/token versions = all
23592025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /device/verify versions = all
23602025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /experimental/v1/probes versions = all
23612025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /experimental/v1/probes versions = all
23622025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /experimental/v1/probes/{probe} versions = all
23632025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /experimental/v1/probes/{probe} versions = all
23642025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /experimental/v1/system/support-bundles versions = all
23652025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /experimental/v1/system/support-bundles versions = all
23662025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23672025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23682025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23692025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23702025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23712025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23722025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
23732025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /images versions = all
23742025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /login versions = all
23752025-04-30T06:42:25.865ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /login/{silo_name}/local versions = all
23762025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
23772025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
23782025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
23792025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /lookup/{path} versions = all
23802025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /projects/{path} versions = all
23812025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /projects-new versions = all
23822025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /settings/{path} versions = all
23832025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /system/{path} versions = all
23842025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /utilization versions = all
23852025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/affinity-groups versions = all
23862025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/affinity-groups versions = all
23872025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
23882025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
23892025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
23902025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
23912025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
23922025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
23932025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
23942025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/anti-affinity-groups versions = all
23952025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/anti-affinity-groups versions = all
23962025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
23972025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
23982025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
23992025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24002025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24012025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24022025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24032025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/certificates versions = all
24042025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/certificates versions = all
24052025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/certificates/{certificate} versions = all
24062025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/certificates/{certificate} versions = all
24072025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/disks versions = all
24082025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/disks versions = all
24092025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/disks/{disk} versions = all
24102025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/disks/{disk} versions = all
24112025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24122025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24132025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24142025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/disks/{disk}/finalize versions = all
24152025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24162025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/floating-ips versions = all
24172025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/floating-ips versions = all
24182025-04-30T06:42:25.866ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24192025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24202025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24212025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24222025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24232025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/groups versions = all
24242025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/groups/{group_id} versions = all
24252025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/images versions = all
24262025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/images versions = all
24272025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/images/{image} versions = all
24282025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/images/{image} versions = all
24292025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/images/{image}/demote versions = all
24302025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/images/{image}/promote versions = all
24312025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/instances versions = all
24322025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/instances versions = all
24332025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/instances/{instance} versions = all
24342025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/instances/{instance} versions = all
24352025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/instances/{instance} versions = all
24362025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24372025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24382025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/instances/{instance}/disks versions = all
24392025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24402025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24412025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/instances/{instance}/external-ips versions = all
24422025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24432025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24442025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/instances/{instance}/reboot versions = all
24452025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/instances/{instance}/serial-console versions = all
24462025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24472025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24482025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/instances/{instance}/start versions = all
24492025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/instances/{instance}/stop versions = all
24502025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/internet-gateway-ip-addresses versions = all
24512025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/internet-gateway-ip-addresses versions = all
24522025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
24532025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/internet-gateway-ip-pools versions = all
24542025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/internet-gateway-ip-pools versions = all
24552025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
24562025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/internet-gateways versions = all
24572025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/internet-gateways versions = all
24582025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
24592025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/internet-gateways/{gateway} versions = all
24602025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/ip-pools versions = all
24612025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/ip-pools/{pool} versions = all
24622025-04-30T06:42:25.867ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/login/{silo_name}/local versions = all
24632025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/logout versions = all
24642025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/me versions = all
24652025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/me/groups versions = all
24662025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/me/ssh-keys versions = all
24672025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/me/ssh-keys versions = all
24682025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
24692025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
24702025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/metrics/{metric_name} versions = all
24712025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/network-interfaces versions = all
24722025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/network-interfaces versions = all
24732025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/network-interfaces/{interface} versions = all
24742025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/network-interfaces/{interface} versions = all
24752025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/network-interfaces/{interface} versions = all
24762025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/ping versions = all
24772025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/policy versions = all
24782025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/policy versions = all
24792025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/projects versions = all
24802025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/projects versions = all
24812025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/projects/{project} versions = all
24822025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/projects/{project} versions = all
24832025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/projects/{project} versions = all
24842025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/projects/{project}/policy versions = all
24852025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/projects/{project}/policy versions = all
24862025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/snapshots versions = all
24872025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/snapshots versions = all
24882025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/snapshots/{snapshot} versions = all
24892025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/snapshots/{snapshot} versions = all
24902025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/disks versions = all
24912025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
24922025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
24932025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/racks versions = all
24942025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
24952025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/sleds versions = all
24962025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/hardware/sleds versions = all
24972025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
24982025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
24992025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25002025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25012025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25022025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/switch-port versions = all
25032025-04-30T06:42:25.868ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25042025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25052025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25062025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25072025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25082025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/switches versions = all
25092025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25102025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/identity-providers versions = all
25112025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/identity-providers/local/users versions = all
25122025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25132025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25142025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/identity-providers/saml versions = all
25152025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25162025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/ip-pools versions = all
25172025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/ip-pools versions = all
25182025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25192025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/ip-pools/{pool} versions = all
25202025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25212025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25222025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25232025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25242025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25252025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25262025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25272025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25282025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25292025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/ip-pools-service versions = all
25302025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25312025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25322025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25332025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/metrics/{metric_name} versions = all
25342025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/address-lot versions = all
25352025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/networking/address-lot versions = all
25362025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25372025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25382025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/allow-list versions = all
25392025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/networking/allow-list versions = all
25402025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/networking/bfd-disable versions = all
25412025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/networking/bfd-enable versions = all
25422025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/bfd-status versions = all
25432025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/networking/bgp versions = all
25442025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/bgp versions = all
25452025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/networking/bgp versions = all
25462025-04-30T06:42:25.869ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25472025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25482025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25492025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25502025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/bgp-exported versions = all
25512025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/bgp-message-history versions = all
25522025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
25532025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/bgp-status versions = all
25542025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/loopback-address versions = all
25552025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/networking/loopback-address versions = all
25562025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
25572025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
25582025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/switch-port-settings versions = all
25592025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/networking/switch-port-settings versions = all
25602025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
25612025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/policy versions = all
25622025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/policy versions = all
25632025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/roles versions = all
25642025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/roles/{role_name} versions = all
25652025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/silo-quotas versions = all
25662025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/silos versions = all
25672025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/silos versions = all
25682025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/system/silos/{silo} versions = all
25692025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/silos/{silo} versions = all
25702025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
25712025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/silos/{silo}/policy versions = all
25722025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/silos/{silo}/policy versions = all
25732025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/silos/{silo}/quotas versions = all
25742025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
25752025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/system/timeseries/query versions = all
25762025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/timeseries/schemas versions = all
25772025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/update/repository versions = all
25782025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/update/repository/{system_version} versions = all
25792025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/update/target-release versions = all
25802025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/system/update/target-release versions = all
25812025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/users versions = all
25822025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/users/{user_id} versions = all
25832025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/users-builtin versions = all
25842025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/users-builtin/{user} versions = all
25852025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/utilization/silos versions = all
25862025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/system/utilization/silos/{silo} versions = all
25872025-04-30T06:42:25.870ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/timeseries/query versions = all
25882025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/users versions = all
25892025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/utilization versions = all
25902025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpc-firewall-rules versions = all
25912025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/vpc-firewall-rules versions = all
25922025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpc-router-routes versions = all
25932025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/vpc-router-routes versions = all
25942025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
25952025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpc-router-routes/{route} versions = all
25962025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/vpc-router-routes/{route} versions = all
25972025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpc-routers versions = all
25982025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/vpc-routers versions = all
25992025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/vpc-routers/{router} versions = all
26002025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpc-routers/{router} versions = all
26012025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/vpc-routers/{router} versions = all
26022025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpc-subnets versions = all
26032025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/vpc-subnets versions = all
26042025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26052025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26062025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26072025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26082025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpcs versions = all
26092025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/vpcs versions = all
26102025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/vpcs/{vpc} versions = all
26112025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/vpcs/{vpc} versions = all
26122025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/vpcs/{vpc} versions = all
26132025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/webhooks/deliveries versions = all
26142025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26152025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/webhooks/event-classes versions = all
26162025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/webhooks/receivers versions = all
26172025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/webhooks/receivers versions = all
26182025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26192025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26202025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26212025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26222025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26232025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26242025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = GET path = /v1/webhooks/secrets versions = all
26252025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = POST path = /v1/webhooks/secrets versions = all
26262025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): registered endpoint local_addr = 127.0.0.1:43866 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26272025-04-30T06:42:25.871ZINFOtest_delete_snapshot (dropshot_external): listening local_addr = 127.0.0.1:43866
26282025-04-30T06:42:25.871ZDEBGtest_delete_snapshot (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:43866
26292025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = / versions = all
26302025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /access versions = all
26312025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /assets/{path} versions = all
26322025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /device/auth versions = all
26332025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /device/confirm versions = all
26342025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /device/success versions = all
26352025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /device/token versions = all
26362025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /device/verify versions = all
26372025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /experimental/v1/probes versions = all
26382025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /experimental/v1/probes versions = all
26392025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26402025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /experimental/v1/probes/{probe} versions = all
26412025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /experimental/v1/system/support-bundles versions = all
26422025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /experimental/v1/system/support-bundles versions = all
26432025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26442025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26452025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26462025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26472025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26482025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26492025-04-30T06:42:25.909ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26502025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /images versions = all
26512025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /login versions = all
26522025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /login/{silo_name}/local versions = all
26532025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
26542025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
26552025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
26562025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /lookup/{path} versions = all
26572025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /projects/{path} versions = all
26582025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /projects-new versions = all
26592025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /settings/{path} versions = all
26602025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /system/{path} versions = all
26612025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /utilization versions = all
26622025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/affinity-groups versions = all
26632025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/affinity-groups versions = all
26642025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
26652025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
26662025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
26672025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
26682025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26692025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26702025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26712025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/anti-affinity-groups versions = all
26722025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/anti-affinity-groups versions = all
26732025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26742025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26752025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26762025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
26772025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26782025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26792025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26802025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/certificates versions = all
26812025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/certificates versions = all
26822025-04-30T06:42:25.910ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/certificates/{certificate} versions = all
26832025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/certificates/{certificate} versions = all
26842025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/disks versions = all
26852025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/disks versions = all
26862025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/disks/{disk} versions = all
26872025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/disks/{disk} versions = all
26882025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/disks/{disk}/bulk-write versions = all
26892025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
26902025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
26912025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/disks/{disk}/finalize versions = all
26922025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
26932025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/floating-ips versions = all
26942025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/floating-ips versions = all
26952025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
26962025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/floating-ips/{floating_ip} versions = all
26972025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
26982025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
26992025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27002025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/groups versions = all
27012025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/groups/{group_id} versions = all
27022025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/images versions = all
27032025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/images versions = all
27042025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/images/{image} versions = all
27052025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/images/{image} versions = all
27062025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/images/{image}/demote versions = all
27072025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/images/{image}/promote versions = all
27082025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/instances versions = all
27092025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/instances versions = all
27102025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/instances/{instance} versions = all
27112025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/instances/{instance} versions = all
27122025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/instances/{instance} versions = all
27132025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27142025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27152025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/instances/{instance}/disks versions = all
27162025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27172025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27182025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/instances/{instance}/external-ips versions = all
27192025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27202025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27212025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/instances/{instance}/reboot versions = all
27222025-04-30T06:42:25.911ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/instances/{instance}/serial-console versions = all
27232025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27242025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27252025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/instances/{instance}/start versions = all
27262025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/instances/{instance}/stop versions = all
27272025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27282025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27292025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27302025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/internet-gateway-ip-pools versions = all
27312025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/internet-gateway-ip-pools versions = all
27322025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27332025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/internet-gateways versions = all
27342025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/internet-gateways versions = all
27352025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27362025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/internet-gateways/{gateway} versions = all
27372025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/ip-pools versions = all
27382025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/ip-pools/{pool} versions = all
27392025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/login/{silo_name}/local versions = all
27402025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/logout versions = all
27412025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/me versions = all
27422025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/me/groups versions = all
27432025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/me/ssh-keys versions = all
27442025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/me/ssh-keys versions = all
27452025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27462025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27472025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/metrics/{metric_name} versions = all
27482025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/network-interfaces versions = all
27492025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/network-interfaces versions = all
27502025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/network-interfaces/{interface} versions = all
27512025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/network-interfaces/{interface} versions = all
27522025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/network-interfaces/{interface} versions = all
27532025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/ping versions = all
27542025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/policy versions = all
27552025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/policy versions = all
27562025-04-30T06:42:25.912ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/projects versions = all
27572025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/projects versions = all
27582025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/projects/{project} versions = all
27592025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/projects/{project} versions = all
27602025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/projects/{project} versions = all
27612025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/projects/{project}/policy versions = all
27622025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/projects/{project}/policy versions = all
27632025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/snapshots versions = all
27642025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/snapshots versions = all
27652025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/snapshots/{snapshot} versions = all
27662025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/snapshots/{snapshot} versions = all
27672025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/disks versions = all
27682025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
27692025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
27702025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/racks versions = all
27712025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
27722025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/sleds versions = all
27732025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/hardware/sleds versions = all
27742025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
27752025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
27762025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
27772025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
27782025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
27792025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/switch-port versions = all
27802025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27812025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27822025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
27832025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
27842025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
27852025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/switches versions = all
27862025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
27872025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/identity-providers versions = all
27882025-04-30T06:42:25.913ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/identity-providers/local/users versions = all
27892025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
27902025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
27912025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/identity-providers/saml versions = all
27922025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
27932025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/ip-pools versions = all
27942025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/ip-pools versions = all
27952025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
27962025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/ip-pools/{pool} versions = all
27972025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/ip-pools/{pool} versions = all
27982025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
27992025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28002025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28012025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28022025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28032025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28042025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28052025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28062025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/ip-pools-service versions = all
28072025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28082025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28092025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28102025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/metrics/{metric_name} versions = all
28112025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/address-lot versions = all
28122025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/networking/address-lot versions = all
28132025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28142025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28152025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/allow-list versions = all
28162025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/networking/allow-list versions = all
28172025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/networking/bfd-disable versions = all
28182025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/networking/bfd-enable versions = all
28192025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/bfd-status versions = all
28202025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/networking/bgp versions = all
28212025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/bgp versions = all
28222025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/networking/bgp versions = all
28232025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28242025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28252025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28262025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28272025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/bgp-exported versions = all
28282025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/bgp-message-history versions = all
28292025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28302025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/bgp-status versions = all
28312025-04-30T06:42:25.914ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/loopback-address versions = all
28322025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/networking/loopback-address versions = all
28332025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28342025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28352025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/switch-port-settings versions = all
28362025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/networking/switch-port-settings versions = all
28372025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28382025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/policy versions = all
28392025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/policy versions = all
28402025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/roles versions = all
28412025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/roles/{role_name} versions = all
28422025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/silo-quotas versions = all
28432025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/silos versions = all
28442025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/silos versions = all
28452025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/system/silos/{silo} versions = all
28462025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/silos/{silo} versions = all
28472025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28482025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/silos/{silo}/policy versions = all
28492025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28502025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/silos/{silo}/quotas versions = all
28512025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
28522025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/system/timeseries/query versions = all
28532025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/timeseries/schemas versions = all
28542025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/update/repository versions = all
28552025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/update/repository/{system_version} versions = all
28562025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/update/target-release versions = all
28572025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/system/update/target-release versions = all
28582025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/users versions = all
28592025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/users/{user_id} versions = all
28602025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/users-builtin versions = all
28612025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/users-builtin/{user} versions = all
28622025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/utilization/silos versions = all
28632025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/system/utilization/silos/{silo} versions = all
28642025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/timeseries/query versions = all
28652025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/users versions = all
28662025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/utilization versions = all
28672025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpc-firewall-rules versions = all
28682025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/vpc-firewall-rules versions = all
28692025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpc-router-routes versions = all
28702025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/vpc-router-routes versions = all
28712025-04-30T06:42:25.915ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
28722025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpc-router-routes/{route} versions = all
28732025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/vpc-router-routes/{route} versions = all
28742025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpc-routers versions = all
28752025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/vpc-routers versions = all
28762025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/vpc-routers/{router} versions = all
28772025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpc-routers/{router} versions = all
28782025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/vpc-routers/{router} versions = all
28792025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpc-subnets versions = all
28802025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/vpc-subnets versions = all
28812025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
28822025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpc-subnets/{subnet} versions = all
28832025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
28842025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
28852025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpcs versions = all
28862025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/vpcs versions = all
28872025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/vpcs/{vpc} versions = all
28882025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/vpcs/{vpc} versions = all
28892025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/vpcs/{vpc} versions = all
28902025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/webhooks/deliveries versions = all
28912025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
28922025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/webhooks/event-classes versions = all
28932025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/webhooks/receivers versions = all
28942025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/webhooks/receivers versions = all
28952025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
28962025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
28972025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
28982025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
28992025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29002025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29012025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = GET path = /v1/webhooks/secrets versions = all
29022025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = POST path = /v1/webhooks/secrets versions = all
29032025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:50384 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29042025-04-30T06:42:25.916ZINFOtest_delete_snapshot (dropshot_external_techport): listening local_addr = 127.0.0.1:50384
29052025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:50384
29062025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (nexus-producer-server): registered DTrace probes
29072025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = 127.0.0.1:65039 method = GET path = /{producer_id} versions = all
29082025-04-30T06:42:25.916ZINFOtest_delete_snapshot (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:65039
29092025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:65039
29102025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:65039
29112025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:35404
29122025-04-30T06:42:25.916ZDEBGtest_delete_snapshot (nexus-producer-server): starting producer registration task
29132025-04-30T06:42:25.916ZINFOtest_delete_snapshot (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:65039 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
29142025-04-30T06:42:25.917ZDEBGtest_delete_snapshot (init_with_steps): Running step start_oximeter
29152025-04-30T06:42:25.917ZDEBGtest_delete_snapshot: Starting Oximeter
29162025-04-30T06:42:25.917ZDEBGtest_delete_snapshot (oximeter): registered DTrace probes
29172025-04-30T06:42:25.917ZINFOtest_delete_snapshot (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29182025-04-30T06:42:25.917ZDEBGtest_delete_snapshot (oximeter): creating ClickHouse client
29192025-04-30T06:42:25.917ZDEBGtest_delete_snapshot (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
29202025-04-30T06:42:25.917ZDEBGtest_delete_snapshot (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29212025-04-30T06:42:25.917ZDEBGtest_delete_snapshot (producer-registration-task): will register with Nexus at 127.0.0.1:35404
29222025-04-30T06:42:25.959ZDEBGtest_delete_snapshot (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:35404/metrics/producers
29232025-04-30T06:42:25.960ZINFOtest_delete_snapshot (dropshot_internal): accepted connection local_addr = 127.0.0.1:35404 remote_addr = 127.0.0.1:41571
29242025-04-30T06:42:25.962ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41571 req_id = db5fc506-a6a9-4e27-8e57-b2e975272c65 uri = /metrics/producers
29252025-04-30T06:42:25.964ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41571 req_id = db5fc506-a6a9-4e27-8e57-b2e975272c65 resource = Database uri = /metrics/producers
29262025-04-30T06:42:25.964ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41571 req_id = db5fc506-a6a9-4e27-8e57-b2e975272c65 roles = RoleSet { roles: {} } uri = /metrics/producers
29272025-04-30T06:42:25.964ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41571 req_id = db5fc506-a6a9-4e27-8e57-b2e975272c65 resource = Database result = Ok(()) uri = /metrics/producers
29282025-04-30T06:42:25.967ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29292025-04-30T06:42:25.969ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bff32071-96df-43c9-8f24-11b42954cf87", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
29302025-04-30T06:42:25.969ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
29312025-04-30T06:42:25.975ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19208220-1a80-4103-ad5f-94ef92770825", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
29322025-04-30T06:42:25.975ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
29332025-04-30T06:42:25.979ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64bb17c1-1074-426d-802f-b519d83a6569", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
29342025-04-30T06:42:25.980ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
29352025-04-30T06:42:25.980ZWARNtest_delete_snapshot (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
29362025-04-30T06:42:25.980ZDEBGtest_delete_snapshot (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29372025-04-30T06:42:25.980ZINFOtest_delete_snapshot (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
29382025-04-30T06:42:25.981ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29392025-04-30T06:42:25.982ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65d8d0a8-1009-4fa5-9662-904097d5cf90", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
29402025-04-30T06:42:25.982ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
29412025-04-30T06:42:25.983ZWARNtest_delete_snapshot (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
29422025-04-30T06:42:25.983ZINFOtest_delete_snapshot (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 version = 0
29432025-04-30T06:42:25.983ZINFOtest_delete_snapshot (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
29442025-04-30T06:42:25.983ZDEBGtest_delete_snapshot (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
29452025-04-30T06:42:25.984ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = CREATE DATABASE IF NOT EXISTS oximeter
29462025-04-30T06:42:25.986ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
29472025-04-30T06:42:25.986ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e143749e-14bc-4765-a687-5d443cb2dad9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
29482025-04-30T06:42:25.987ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
29492025-04-30T06:42:25.988ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a00977b3-3973-447b-9af8-8ac3fc21dbf2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
29502025-04-30T06:42:25.989ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
29512025-04-30T06:42:25.989ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29522025-04-30T06:42:25.990ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "701df73b-73a2-4ed8-91ec-732bc3be3774", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
29532025-04-30T06:42:25.991ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29542025-04-30T06:42:25.991ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29552025-04-30T06:42:25.991ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29562025-04-30T06:42:25.991ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29572025-04-30T06:42:25.992ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29582025-04-30T06:42:25.993ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29592025-04-30T06:42:25.996ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29602025-04-30T06:42:25.999ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29612025-04-30T06:42:25.999ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29622025-04-30T06:42:25.999ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29632025-04-30T06:42:25.999ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29642025-04-30T06:42:25.999ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29652025-04-30T06:42:26.001ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29662025-04-30T06:42:26.002ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29672025-04-30T06:42:26.007ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29682025-04-30T06:42:26.008ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29692025-04-30T06:42:26.014ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29702025-04-30T06:42:26.014ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29712025-04-30T06:42:26.014ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29722025-04-30T06:42:26.014ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29732025-04-30T06:42:26.015ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29742025-04-30T06:42:26.022ZINFOtest_delete_snapshot (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 60053 local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41571 req_id = db5fc506-a6a9-4e27-8e57-b2e975272c65 response_code = 503 uri = /metrics/producers
29752025-04-30T06:42:26.072ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
29762025-04-30T06:42:26.072ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
29772025-04-30T06:42:26.072ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29782025-04-30T06:42:26.074ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:38320 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29792025-04-30T06:42:26.075ZDEBGtest_delete_snapshot (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "db5fc506-a6a9-4e27-8e57-b2e975272c65", "content-length": "133", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
29802025-04-30T06:42:26.077ZWARNtest_delete_snapshot (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 160.857779ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"db5fc506-a6a9-4e27-8e57-b2e975272c65\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:42:25 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"db5fc506-a6a9-4e27-8e57-b2e975272c65\\" }" file = oximeter/producer/src/lib.rs:371
29812025-04-30T06:42:26.079ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29822025-04-30T06:42:26.079ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:38320 req_id = 9f6a1b9f-998a-4904-a607-d17359c5af52 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29832025-04-30T06:42:26.080ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 88 d1 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:59078 req_id = 12e34876-16cf-4df3-8c37-9b34ee73dd77
29842025-04-30T06:42:26.080ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35025,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59078 req_id = 12e34876-16cf-4df3-8c37-9b34ee73dd77
29852025-04-30T06:42:26.080ZDEBGtest_delete_snapshot (store): query key key = _crucible-pantry._tcp
29862025-04-30T06:42:26.080ZDEBGtest_delete_snapshot (store): query key key = 7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host
29872025-04-30T06:42:26.080ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59078 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 33598, target: Name("7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host.control-plane.oxide.internal") })) }] req_id = 12e34876-16cf-4df3-8c37-9b34ee73dd77
29882025-04-30T06:42:26.081ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1703 local_addr = [::1]:62607 method = GET remote_addr = [::1]:38320 req_id = 9f6a1b9f-998a-4904-a607-d17359c5af52 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29892025-04-30T06:42:26.083ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f6a1b9f-998a-4904-a607-d17359c5af52", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
29902025-04-30T06:42:26.085ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29912025-04-30T06:42:26.085ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29922025-04-30T06:42:26.085ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
29932025-04-30T06:42:26.087ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
29942025-04-30T06:42:26.087ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29952025-04-30T06:42:26.087ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29962025-04-30T06:42:26.088ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29972025-04-30T06:42:26.088ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29982025-04-30T06:42:26.089ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29992025-04-30T06:42:26.093ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 16 32 01 00 00 01 00 00 00 00 00 01 24 37 63 62 .2..........$7cb\n0010: 38 65 64 35 32 2d 66 35 63 38 2d 34 36 61 31 2d 8ed52-f5c8-46a1-\n0020: 61 38 64 64 2d 65 35 63 62 39 66 34 39 66 63 35 a8dd-e5cb9f49fc5\n0030: 61 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 a.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:64949 req_id = 8ce3ea9b-a30d-4e53-848b-2028544feb37
30002025-04-30T06:42:26.093ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5682,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 55,\n 99,\n 98,\n 56,\n 101,\n 100,\n 53,\n 50,\n 45,\n 102,\n 53,\n 99,\n 56,\n 45,\n 52,\n 54,\n 97,\n 49,\n 45,\n 97,\n 56,\n 100,\n 100,\n 45,\n 101,\n 53,\n 99,\n 98,\n 57,\n 102,\n 52,\n 57,\n 102,\n 99,\n 53,\n 97,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64949 req_id = 8ce3ea9b-a30d-4e53-848b-2028544feb37
30012025-04-30T06:42:26.093ZDEBGtest_delete_snapshot (store): query key key = 7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host
30022025-04-30T06:42:26.093ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64949 query = LowerQuery { name: LowerName(Name("7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host.control-plane.oxide.internal.")), original: Query { name: Name("7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("7cb8ed52-f5c8-46a1-a8dd-e5cb9f49fc5a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8ce3ea9b-a30d-4e53-848b-2028544feb37
30032025-04-30T06:42:26.093ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30042025-04-30T06:42:26.101ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
30052025-04-30T06:42:26.101ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30062025-04-30T06:42:26.101ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30072025-04-30T06:42:26.102ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30082025-04-30T06:42:26.102ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30092025-04-30T06:42:26.167ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30102025-04-30T06:42:26.171ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:54600 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30112025-04-30T06:42:26.175ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30122025-04-30T06:42:26.180ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:54600 req_id = ad96bdb4-8b5b-4adf-9c37-c5d43e490d2f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30132025-04-30T06:42:26.181ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 581 local_addr = [::1]:33598 method = GET remote_addr = [::1]:54600 req_id = ad96bdb4-8b5b-4adf-9c37-c5d43e490d2f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30142025-04-30T06:42:26.181ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30152025-04-30T06:42:26.652ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30162025-04-30T06:42:26.656ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:36754 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30172025-04-30T06:42:26.656ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:33919 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30182025-04-30T06:42:26.656ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:37783 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30192025-04-30T06:42:26.656ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:46291 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30202025-04-30T06:42:26.656ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:50323 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30212025-04-30T06:42:26.656ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:43227 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30222025-04-30T06:42:26.657ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:45239 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30232025-04-30T06:42:26.658ZDEBGtest_delete_snapshot (producer-registration-task): will register with Nexus at 127.0.0.1:35404
30242025-04-30T06:42:26.723ZDEBGtest_delete_snapshot (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:35404/metrics/producers
30252025-04-30T06:42:26.725ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30262025-04-30T06:42:26.725ZINFOtest_delete_snapshot (dropshot_internal): accepted connection local_addr = 127.0.0.1:35404 remote_addr = 127.0.0.1:65302
30272025-04-30T06:42:26.726ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:36754 req_id = c3cf2b4d-1773-4287-b567-e7492ea1ba1a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30282025-04-30T06:42:26.726ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:33919 req_id = 80f9fca0-a2f6-4995-980b-a9fe36c90388 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30292025-04-30T06:42:26.726ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:37783 req_id = 52eac1a9-cde8-4dd4-a5a2-854a07fdce85 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30302025-04-30T06:42:26.726ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:46291 req_id = 65b0528c-8f24-4b37-80ec-1b57af48c656 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30312025-04-30T06:42:26.726ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:50323 req_id = 0c2ec2a5-d68e-4b21-bb85-0f1b6a1aeb7e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30322025-04-30T06:42:26.727ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:43227 req_id = aa73ff37-b5df-4186-a2cf-e978b5de3faa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30332025-04-30T06:42:26.727ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:45239 req_id = 4d811dd4-a7db-4048-b511-e9f1a5b71111 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30342025-04-30T06:42:26.740ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 14036 local_addr = [::1]:33598 method = GET remote_addr = [::1]:36754 req_id = c3cf2b4d-1773-4287-b567-e7492ea1ba1a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30352025-04-30T06:42:26.740ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 13929 local_addr = [::1]:33598 method = GET remote_addr = [::1]:33919 req_id = 80f9fca0-a2f6-4995-980b-a9fe36c90388 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30362025-04-30T06:42:26.740ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 13850 local_addr = [::1]:33598 method = GET remote_addr = [::1]:37783 req_id = 52eac1a9-cde8-4dd4-a5a2-854a07fdce85 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30372025-04-30T06:42:26.740ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 13531 local_addr = [::1]:33598 method = GET remote_addr = [::1]:46291 req_id = 65b0528c-8f24-4b37-80ec-1b57af48c656 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30382025-04-30T06:42:26.740ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 13435 local_addr = [::1]:33598 method = GET remote_addr = [::1]:50323 req_id = 0c2ec2a5-d68e-4b21-bb85-0f1b6a1aeb7e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30392025-04-30T06:42:26.740ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 13354 local_addr = [::1]:33598 method = GET remote_addr = [::1]:43227 req_id = aa73ff37-b5df-4186-a2cf-e978b5de3faa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30402025-04-30T06:42:26.740ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 13280 local_addr = [::1]:33598 method = GET remote_addr = [::1]:45239 req_id = 4d811dd4-a7db-4048-b511-e9f1a5b71111 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30412025-04-30T06:42:26.740ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30422025-04-30T06:42:26.742ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:65302 req_id = cdaf9738-492d-4622-a287-d26c40f838e5 uri = /metrics/producers
30432025-04-30T06:42:26.743ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:65302 req_id = cdaf9738-492d-4622-a287-d26c40f838e5 resource = Database uri = /metrics/producers
30442025-04-30T06:42:26.743ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:65302 req_id = cdaf9738-492d-4622-a287-d26c40f838e5 roles = RoleSet { roles: {} } uri = /metrics/producers
30452025-04-30T06:42:26.744ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:65302 req_id = cdaf9738-492d-4622-a287-d26c40f838e5 resource = Database result = Ok(()) uri = /metrics/producers
30462025-04-30T06:42:26.744ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30472025-04-30T06:42:26.751ZINFOtest_delete_snapshot (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 8856 local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:65302 req_id = cdaf9738-492d-4622-a287-d26c40f838e5 response_code = 503 uri = /metrics/producers
30482025-04-30T06:42:26.751ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30492025-04-30T06:42:26.752ZDEBGtest_delete_snapshot (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "cdaf9738-492d-4622-a287-d26c40f838e5", "content-length": "133", "date": "Wed, 30 Apr 2025 06:42:25 GMT"} })
30502025-04-30T06:42:26.753ZWARNtest_delete_snapshot (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 837.264025ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"cdaf9738-492d-4622-a287-d26c40f838e5\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:42:25 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"cdaf9738-492d-4622-a287-d26c40f838e5\\" }" file = oximeter/producer/src/lib.rs:371
30512025-04-30T06:42:26.756ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30522025-04-30T06:42:26.760ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30532025-04-30T06:42:26.766ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30542025-04-30T06:42:26.773ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30552025-04-30T06:42:26.778ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30562025-04-30T06:42:26.781ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30572025-04-30T06:42:26.785ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30582025-04-30T06:42:26.789ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30592025-04-30T06:42:26.793ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30602025-04-30T06:42:26.797ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30612025-04-30T06:42:26.803ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30622025-04-30T06:42:26.810ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30632025-04-30T06:42:26.824ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30642025-04-30T06:42:26.828ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30652025-04-30T06:42:26.828ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
30662025-04-30T06:42:26.829ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
30672025-04-30T06:42:26.829ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.884159575s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
30682025-04-30T06:42:26.831ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30692025-04-30T06:42:26.842ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30702025-04-30T06:42:26.852ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30712025-04-30T06:42:26.855ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30722025-04-30T06:42:26.860ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30732025-04-30T06:42:26.867ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30742025-04-30T06:42:26.871ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30752025-04-30T06:42:26.876ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30762025-04-30T06:42:26.880ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30772025-04-30T06:42:26.886ZINFOtest_delete_snapshot (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 version = 14
30782025-04-30T06:42:26.886ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30792025-04-30T06:42:26.893ZDEBGtest_delete_snapshot (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8a5bc354-ea09-45fa-a5db-a26f4f663208
30802025-04-30T06:42:26.898ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:57460 method = GET path = /info versions = all
30812025-04-30T06:42:26.898ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:57460 method = GET path = /producers versions = all
30822025-04-30T06:42:26.898ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:57460 method = DELETE path = /producers/{producer_id} versions = all
30832025-04-30T06:42:26.898ZDEBGtest_delete_snapshot (dropshot): registered endpoint local_addr = [::1]:57460 method = GET path = /producers/{producer_id} versions = all
30842025-04-30T06:42:26.898ZINFOtest_delete_snapshot (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:57460
30852025-04-30T06:42:26.899ZDEBGtest_delete_snapshot (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:57460
30862025-04-30T06:42:26.900ZDEBGtest_delete_snapshot (oximeter): registered USDT probes
30872025-04-30T06:42:26.900ZDEBGtest_delete_snapshot (oximeter): contacting nexus
30882025-04-30T06:42:27.411ZDEBGtest_delete_snapshot (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:35404/metrics/collectors
30892025-04-30T06:42:27.411ZDEBGtest_delete_snapshot (producer-registration-task): will register with Nexus at 127.0.0.1:35404
30902025-04-30T06:42:27.463ZDEBGtest_delete_snapshot (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:35404/metrics/producers
30912025-04-30T06:42:27.464ZINFOtest_delete_snapshot (dropshot_internal): accepted connection local_addr = 127.0.0.1:35404 remote_addr = 127.0.0.1:41747
30922025-04-30T06:42:27.464ZINFOtest_delete_snapshot (dropshot_internal): accepted connection local_addr = 127.0.0.1:35404 remote_addr = 127.0.0.1:54377
30932025-04-30T06:42:27.465ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:54377 req_id = 956a737e-2cbc-40a7-a596-ee409b8356fe uri = /metrics/producers
30942025-04-30T06:42:27.465ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:54377 req_id = 956a737e-2cbc-40a7-a596-ee409b8356fe resource = Database uri = /metrics/producers
30952025-04-30T06:42:27.465ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:54377 req_id = 956a737e-2cbc-40a7-a596-ee409b8356fe roles = RoleSet { roles: {} } uri = /metrics/producers
30962025-04-30T06:42:27.466ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:54377 req_id = 956a737e-2cbc-40a7-a596-ee409b8356fe resource = Database result = Ok(()) uri = /metrics/producers
30972025-04-30T06:42:27.468ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41747 req_id = 121fc15e-0fa8-404b-bee3-26001e5709f2 uri = /metrics/collectors
30982025-04-30T06:42:27.468ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41747 req_id = 121fc15e-0fa8-404b-bee3-26001e5709f2 resource = Database uri = /metrics/collectors
30992025-04-30T06:42:27.468ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41747 req_id = 121fc15e-0fa8-404b-bee3-26001e5709f2 roles = RoleSet { roles: {} } uri = /metrics/collectors
31002025-04-30T06:42:27.469ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41747 req_id = 121fc15e-0fa8-404b-bee3-26001e5709f2 resource = Database result = Ok(()) uri = /metrics/collectors
31012025-04-30T06:42:27.473ZINFOtest_delete_snapshot (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 8046 local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:54377 req_id = 956a737e-2cbc-40a7-a596-ee409b8356fe response_code = 503 uri = /metrics/producers
31022025-04-30T06:42:27.473ZDEBGtest_delete_snapshot (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "956a737e-2cbc-40a7-a596-ee409b8356fe", "content-length": "133", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
31032025-04-30T06:42:27.475ZWARNtest_delete_snapshot (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 1.558552023s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"956a737e-2cbc-40a7-a596-ee409b8356fe\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:42:27 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"956a737e-2cbc-40a7-a596-ee409b8356fe\\" }" file = oximeter/producer/src/lib.rs:371
31042025-04-30T06:42:27.477ZINFOtest_delete_snapshot (nexus): registered new oximeter metric collection server address = [::1]:57460 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31052025-04-30T06:42:27.477ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 9742 local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:41747 req_id = 121fc15e-0fa8-404b-bee3-26001e5709f2 response_code = 204 uri = /metrics/collectors
31062025-04-30T06:42:27.478ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors", status: 204, headers: {"x-request-id": "121fc15e-0fa8-404b-bee3-26001e5709f2", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
31072025-04-30T06:42:27.478ZINFOtest_delete_snapshot (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31082025-04-30T06:42:27.478ZDEBGtest_delete_snapshot (init_with_steps): Running step start_producer_server
31092025-04-30T06:42:27.478ZDEBGtest_delete_snapshot: Starting test metric Producer Server
31102025-04-30T06:42:27.484ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64878 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31112025-04-30T06:42:27.485ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64878 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31122025-04-30T06:42:27.485ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64878 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31132025-04-30T06:42:27.485ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64878 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31142025-04-30T06:42:27.485ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64878 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31152025-04-30T06:42:27.485ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64878 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31162025-04-30T06:42:27.485ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64878 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31172025-04-30T06:42:27.485ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64878 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31182025-04-30T06:42:27.485ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:64878 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31192025-04-30T06:42:27.485ZINFOtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:64878 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31202025-04-30T06:42:27.485ZDEBGtest_delete_snapshot (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:64878 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31212025-04-30T06:42:27.485ZINFOtest_delete_snapshot (SledAgent): Created Simulated Crucible Server address = [::1]:64878 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31222025-04-30T06:42:27.486ZINFOtest_delete_snapshot (nexus): inserting test physical disk model = test-model physical_disk_id = 98ebbd6f-544a-488f-9d2a-6e106c37bb13 serial = totally-unique-serial: 98ebbd6f-544a-488f-9d2a-6e106c37bb13 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31232025-04-30T06:42:27.486ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31242025-04-30T06:42:27.486ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31252025-04-30T06:42:27.487ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31262025-04-30T06:42:27.488ZINFOtest_delete_snapshot (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31272025-04-30T06:42:27.546ZINFOtest_delete_snapshot (dropshot_internal): accepted connection local_addr = 127.0.0.1:35404 remote_addr = 127.0.0.1:36430
31282025-04-30T06:42:27.547ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31292025-04-30T06:42:27.547ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31302025-04-30T06:42:27.548ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ea23f459-57e6-40bb-85b6-fee286ae4acb uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31312025-04-30T06:42:27.548ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:36430 req_id = c5c5d077-79a0-4266-842e-a0d95d141c2e uri = /metrics/producers
31322025-04-30T06:42:27.551ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ea23f459-57e6-40bb-85b6-fee286ae4acb resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31332025-04-30T06:42:27.551ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ea23f459-57e6-40bb-85b6-fee286ae4acb roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31342025-04-30T06:42:27.552ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ea23f459-57e6-40bb-85b6-fee286ae4acb resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31352025-04-30T06:42:27.553ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:36430 req_id = c5c5d077-79a0-4266-842e-a0d95d141c2e resource = Database uri = /metrics/producers
31362025-04-30T06:42:27.553ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:36430 req_id = c5c5d077-79a0-4266-842e-a0d95d141c2e roles = RoleSet { roles: {} } uri = /metrics/producers
31372025-04-30T06:42:27.553ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:36430 req_id = c5c5d077-79a0-4266-842e-a0d95d141c2e resource = Database result = Ok(()) uri = /metrics/producers
31382025-04-30T06:42:27.554ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31392025-04-30T06:42:27.554ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31402025-04-30T06:42:27.554ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31412025-04-30T06:42:27.561ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 12927 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ea23f459-57e6-40bb-85b6-fee286ae4acb response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31422025-04-30T06:42:27.561ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31432025-04-30T06:42:27.561ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31442025-04-30T06:42:27.561ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31452025-04-30T06:42:27.562ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31462025-04-30T06:42:27.562ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31472025-04-30T06:42:27.562ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea23f459-57e6-40bb-85b6-fee286ae4acb", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
31482025-04-30T06:42:27.563ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
31492025-04-30T06:42:27.569ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31502025-04-30T06:42:27.573ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31512025-04-30T06:42:27.606ZINFOtest_delete_snapshot (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
31522025-04-30T06:42:27.612ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 63540 local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:36430 req_id = c5c5d077-79a0-4266-842e-a0d95d141c2e response_code = 201 uri = /metrics/producers
31532025-04-30T06:42:27.720ZINFOtest_delete_snapshot (nexus): upserting zpool physical_disk_id = 98ebbd6f-544a-488f-9d2a-6e106c37bb13 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 7966ac54-bc60-4e07-b456-cb4de4fdc081
31542025-04-30T06:42:27.722ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31552025-04-30T06:42:27.722ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31562025-04-30T06:42:27.723ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31572025-04-30T06:42:27.727ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31582025-04-30T06:42:27.727ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31592025-04-30T06:42:27.727ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31602025-04-30T06:42:27.727ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31612025-04-30T06:42:27.727ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31622025-04-30T06:42:27.729ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31632025-04-30T06:42:27.736ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31642025-04-30T06:42:27.786ZINFOtest_delete_snapshot (nexus): upserting Crucible dataset dataset_id = 82d6178d-5495-4fa6-9f93-9f6c1f92aefb zpool_id = 7966ac54-bc60-4e07-b456-cb4de4fdc081
31652025-04-30T06:42:27.798ZINFOtest_delete_snapshot (nexus): inserting test physical disk model = test-model physical_disk_id = 98ebbd6f-544a-488f-9d2a-6e106c37bb13 serial = totally-unique-serial: 98ebbd6f-544a-488f-9d2a-6e106c37bb13 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31662025-04-30T06:42:27.798ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31672025-04-30T06:42:27.798ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31682025-04-30T06:42:27.803ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31692025-04-30T06:42:27.819ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 98ebbd6f-544a-488f-9d2a-6e106c37bb13 (physical_disk), lookup_type: ById(98ebbd6f-544a-488f-9d2a-6e106c37bb13) }
31702025-04-30T06:42:27.819ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31712025-04-30T06:42:27.819ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31722025-04-30T06:42:27.819ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31732025-04-30T06:42:27.820ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31742025-04-30T06:42:27.825ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31752025-04-30T06:42:27.833ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 98ebbd6f-544a-488f-9d2a-6e106c37bb13 (physical_disk), lookup_type: ById(98ebbd6f-544a-488f-9d2a-6e106c37bb13) } result = Ok(())
31762025-04-30T06:42:27.833ZINFOtest_delete_snapshot (nexus): upserting zpool physical_disk_id = 98ebbd6f-544a-488f-9d2a-6e106c37bb13 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 7966ac54-bc60-4e07-b456-cb4de4fdc081
31772025-04-30T06:42:27.833ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31782025-04-30T06:42:27.833ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31792025-04-30T06:42:27.834ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31802025-04-30T06:42:27.836ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31812025-04-30T06:42:27.836ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31822025-04-30T06:42:27.836ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31832025-04-30T06:42:27.836ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31842025-04-30T06:42:27.836ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31852025-04-30T06:42:27.840ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31862025-04-30T06:42:27.844ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31872025-04-30T06:42:27.850ZTRACtest_delete_snapshot (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31882025-04-30T06:42:27.851ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31892025-04-30T06:42:27.851ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31902025-04-30T06:42:27.851ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31912025-04-30T06:42:27.851ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31922025-04-30T06:42:27.858ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31932025-04-30T06:42:27.862ZDEBGtest_delete_snapshot (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31942025-04-30T06:42:27.862ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31952025-04-30T06:42:27.862ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
31962025-04-30T06:42:27.863ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31972025-04-30T06:42:27.872ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
31982025-04-30T06:42:27.872ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31992025-04-30T06:42:27.872ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32002025-04-30T06:42:27.872ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
32012025-04-30T06:42:27.874ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32022025-04-30T06:42:27.874ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
32032025-04-30T06:42:27.874ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
32042025-04-30T06:42:27.874ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32052025-04-30T06:42:27.874ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32062025-04-30T06:42:27.874ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32072025-04-30T06:42:27.876ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32082025-04-30T06:42:27.879ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32092025-04-30T06:42:27.888ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32102025-04-30T06:42:27.888ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32112025-04-30T06:42:27.889ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
32122025-04-30T06:42:27.889ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32132025-04-30T06:42:27.890ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32142025-04-30T06:42:27.896ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
32152025-04-30T06:42:27.896ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32162025-04-30T06:42:27.896ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32172025-04-30T06:42:27.897ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32182025-04-30T06:42:27.905ZTRACtest_delete_snapshot (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32192025-04-30T06:42:27.905ZTRACtest_delete_snapshot (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32202025-04-30T06:42:27.910ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32212025-04-30T06:42:27.910ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
32222025-04-30T06:42:27.910ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32232025-04-30T06:42:27.911ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
32242025-04-30T06:42:27.911ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
32252025-04-30T06:42:27.911ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d8 52 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .R..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51921 req_id = 5bbc8af8-1ec5-45d1-9f1b-7a4f61699b77
32262025-04-30T06:42:27.911ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55378,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51921 req_id = 5bbc8af8-1ec5-45d1-9f1b-7a4f61699b77
32272025-04-30T06:42:27.912ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
32282025-04-30T06:42:27.912ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32292025-04-30T06:42:27.912ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51921 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5bbc8af8-1ec5-45d1-9f1b-7a4f61699b77
32302025-04-30T06:42:27.913ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4212, tv_nsec: 711735662 } })
32312025-04-30T06:42:27.913ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
32322025-04-30T06:42:27.913ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ce af 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56749 req_id = f8178453-44d6-4678-8587-9d77dcf9c37b
32332025-04-30T06:42:27.913ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52911,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56749 req_id = f8178453-44d6-4678-8587-9d77dcf9c37b
32342025-04-30T06:42:27.914ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32352025-04-30T06:42:27.914ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56749 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f8178453-44d6-4678-8587-9d77dcf9c37b
32362025-04-30T06:42:27.986ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
32372025-04-30T06:42:27.987ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8a 72 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .r..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63518 req_id = ba1b6f25-8b8a-48cc-b7a7-e5016db58b24
32382025-04-30T06:42:27.987ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35442,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63518 req_id = ba1b6f25-8b8a-48cc-b7a7-e5016db58b24
32392025-04-30T06:42:27.988ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
32402025-04-30T06:42:27.988ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35442, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63518 req_id = ba1b6f25-8b8a-48cc-b7a7-e5016db58b24
32412025-04-30T06:42:27.988ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
32422025-04-30T06:42:27.988ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
32432025-04-30T06:42:27.988ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
32442025-04-30T06:42:27.997ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9026060-7590-4f15-b2f7-9ac5f7c9b170", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32452025-04-30T06:42:28.006ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
32462025-04-30T06:42:28.008ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75ce48f4-e7ac-4e8e-97bf-ce479154e6e5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32472025-04-30T06:42:28.008ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
32482025-04-30T06:42:28.010ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17fffc8d-2acf-40cd-9173-3e9f8b843432", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32492025-04-30T06:42:28.010ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
32502025-04-30T06:42:28.014ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c01b1df9-ccaf-4c42-abd0-0fee93ac1095", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32512025-04-30T06:42:28.015ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
32522025-04-30T06:42:28.019ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf9a74dc-a920-4702-9e0e-ebe6205044c5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32532025-04-30T06:42:28.019ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
32542025-04-30T06:42:28.023ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2005a401-40d6-4de9-82ed-0949b95983ca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32552025-04-30T06:42:28.024ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
32562025-04-30T06:42:28.025ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb21b601-ca91-4273-89c1-28960d085b72", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32572025-04-30T06:42:28.025ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
32582025-04-30T06:42:28.028ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8a953b1-d305-4cf2-b870-51fecb587d03", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32592025-04-30T06:42:28.028ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
32602025-04-30T06:42:28.031ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b2fdae8-15e4-4df2-98f5-9960fb4e321a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32612025-04-30T06:42:28.032ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
32622025-04-30T06:42:28.033ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7753c896-332a-4875-88b3-fdebdbd2eff7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32632025-04-30T06:42:28.033ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
32642025-04-30T06:42:28.035ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "546922e7-974a-42b8-b81d-d33609535ded", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32652025-04-30T06:42:28.035ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
32662025-04-30T06:42:28.036ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de28f02b-b563-4fff-bb42-412b5c2ba573", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32672025-04-30T06:42:28.036ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
32682025-04-30T06:42:28.040ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1ba3116-6783-4eda-b075-c38a71dc2c73", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32692025-04-30T06:42:28.041ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
32702025-04-30T06:42:28.044ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "017c1811-4dbb-463a-aa4d-d471fe114568", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32712025-04-30T06:42:28.045ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
32722025-04-30T06:42:28.048ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3105ef7-71fa-4207-9c38-58e5979c2bf3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32732025-04-30T06:42:28.048ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
32742025-04-30T06:42:28.051ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fafb23b6-ad1e-4a67-90fd-ffc2f95f9b6a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32752025-04-30T06:42:28.052ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
32762025-04-30T06:42:28.055ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed36808-80bb-43a2-baf9-e3898e5e89fc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32772025-04-30T06:42:28.055ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
32782025-04-30T06:42:28.061ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bf834d6-fbba-4082-853a-badb4829551f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32792025-04-30T06:42:28.061ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
32802025-04-30T06:42:28.069ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
32812025-04-30T06:42:28.118ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd781add-9410-4618-a64b-6ee9a4925058", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32822025-04-30T06:42:28.118ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
32832025-04-30T06:42:28.120ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
32842025-04-30T06:42:28.120ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32852025-04-30T06:42:28.120ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32862025-04-30T06:42:28.121ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
32872025-04-30T06:42:28.122ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32882025-04-30T06:42:28.126ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22b95497-a06f-4730-a8cf-f0a513bc34fc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32892025-04-30T06:42:28.126ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
32902025-04-30T06:42:28.129ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32912025-04-30T06:42:28.135ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32922025-04-30T06:42:28.136ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32932025-04-30T06:42:28.136ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
32942025-04-30T06:42:28.137ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32952025-04-30T06:42:28.137ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7461594d-2c8e-4363-9c35-8daf34a11a57", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32962025-04-30T06:42:28.138ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
32972025-04-30T06:42:28.140ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa90b479-5b5a-4df3-8031-abf1b99c998e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
32982025-04-30T06:42:28.140ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
32992025-04-30T06:42:28.141ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33002025-04-30T06:42:28.141ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
33012025-04-30T06:42:28.142ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33022025-04-30T06:42:28.143ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c33a557-b695-4758-8129-a58d1b954f61", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33032025-04-30T06:42:28.143ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
33042025-04-30T06:42:28.146ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e7a49aa-2491-4de3-ae1f-3744e1e37416", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33052025-04-30T06:42:28.146ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
33062025-04-30T06:42:28.154ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58a96cf1-af1e-42fa-84bd-df442a8295fb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33072025-04-30T06:42:28.154ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
33082025-04-30T06:42:28.156ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1c1c591-dfd5-4a6e-88f6-f4143d37e71a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33092025-04-30T06:42:28.157ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
33102025-04-30T06:42:28.160ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95d00c1b-17c6-430c-8dfa-03f40819f3fc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33112025-04-30T06:42:28.160ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
33122025-04-30T06:42:28.162ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da153a15-4138-4bb5-8af1-98adc02dafb6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33132025-04-30T06:42:28.162ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
33142025-04-30T06:42:28.165ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a349d41e-ef97-408a-b056-19a8f8208e8f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33152025-04-30T06:42:28.166ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
33162025-04-30T06:42:28.170ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c703767c-b75f-42d8-9d82-b2a4cd8d14e2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33172025-04-30T06:42:28.170ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
33182025-04-30T06:42:28.176ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc9390c7-fb52-401f-9875-e0bd70baa421", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33192025-04-30T06:42:28.176ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
33202025-04-30T06:42:28.178ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15a41bc8-5274-4fbe-9249-da71b9a5616d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33212025-04-30T06:42:28.178ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
33222025-04-30T06:42:28.184ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "925fe1a1-0081-4d34-8857-1c3b71d1e55a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33232025-04-30T06:42:28.185ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
33242025-04-30T06:42:28.186ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b353855-fecc-4809-a578-e2b04ac73fd5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33252025-04-30T06:42:28.187ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
33262025-04-30T06:42:28.190ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ed66f26-7b44-4557-8524-9ec5328cfda2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33272025-04-30T06:42:28.190ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
33282025-04-30T06:42:28.195ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
33292025-04-30T06:42:28.199ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7061c5d8-45be-4693-b362-501294d1fb02", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33302025-04-30T06:42:28.199ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
33312025-04-30T06:42:28.202ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77e70156-9f7a-42e8-8a1f-2a881e4e6bdb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33322025-04-30T06:42:28.202ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
33332025-04-30T06:42:28.204ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1007303-6459-426c-8286-81a60fe5f319", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33342025-04-30T06:42:28.204ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
33352025-04-30T06:42:28.209ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f39af0e-c72a-4df0-b661-bdbb53dfbd6c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33362025-04-30T06:42:28.210ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
33372025-04-30T06:42:28.212ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18c0119d-d312-45d8-a3f0-ed093f518c41", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33382025-04-30T06:42:28.212ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
33392025-04-30T06:42:28.214ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83560083-1d6d-443a-babf-c2cf6a9ed65c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33402025-04-30T06:42:28.215ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
33412025-04-30T06:42:28.216ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9db546b0-b22c-4b9a-9717-6b5993cb48c6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33422025-04-30T06:42:28.216ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
33432025-04-30T06:42:28.217ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c3245c1-0f12-433e-a12e-32aafcde1108", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33442025-04-30T06:42:28.217ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
33452025-04-30T06:42:28.218ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aabdb707-0945-48ed-9047-c0db9ff03632", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33462025-04-30T06:42:28.218ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
33472025-04-30T06:42:28.220ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16161e50-103d-4b39-b5d4-af97ed1b52a4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33482025-04-30T06:42:28.221ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33492025-04-30T06:42:28.221ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33502025-04-30T06:42:28.221ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33512025-04-30T06:42:28.221ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33522025-04-30T06:42:28.222ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33532025-04-30T06:42:28.228ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33542025-04-30T06:42:28.232ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33552025-04-30T06:42:28.232ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33562025-04-30T06:42:28.232ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33572025-04-30T06:42:28.232ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33582025-04-30T06:42:28.232ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33592025-04-30T06:42:28.232ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33602025-04-30T06:42:28.235ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33612025-04-30T06:42:28.240ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33622025-04-30T06:42:28.240ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33632025-04-30T06:42:28.240ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33642025-04-30T06:42:28.240ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33652025-04-30T06:42:28.294ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
33662025-04-30T06:42:28.295ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
33672025-04-30T06:42:28.295ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33682025-04-30T06:42:28.295ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33692025-04-30T06:42:28.295ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33702025-04-30T06:42:28.295ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
33712025-04-30T06:42:28.296ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33722025-04-30T06:42:28.296ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:38847 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33732025-04-30T06:42:28.297ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:38847 req_id = a535add2-48be-401c-917b-e024d8117e33 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33742025-04-30T06:42:28.298ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 779 local_addr = [::1]:62607 method = GET remote_addr = [::1]:38847 req_id = a535add2-48be-401c-917b-e024d8117e33 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33752025-04-30T06:42:28.298ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33762025-04-30T06:42:28.304ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33772025-04-30T06:42:28.304ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33782025-04-30T06:42:28.304ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
33792025-04-30T06:42:28.305ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33802025-04-30T06:42:28.306ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a535add2-48be-401c-917b-e024d8117e33", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:27 GMT"} })
33812025-04-30T06:42:28.308ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33822025-04-30T06:42:28.308ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33832025-04-30T06:42:28.308ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
33842025-04-30T06:42:28.309ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
33852025-04-30T06:42:28.309ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33862025-04-30T06:42:28.310ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33872025-04-30T06:42:28.310ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33882025-04-30T06:42:28.310ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33892025-04-30T06:42:28.314ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33902025-04-30T06:42:28.314ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
33912025-04-30T06:42:28.314ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33922025-04-30T06:42:28.314ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33932025-04-30T06:42:28.318ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
33942025-04-30T06:42:28.318ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33952025-04-30T06:42:28.318ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33962025-04-30T06:42:28.319ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33972025-04-30T06:42:28.386ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
33982025-04-30T06:42:28.441ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33992025-04-30T06:42:28.441ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34002025-04-30T06:42:28.441ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34012025-04-30T06:42:28.441ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
34022025-04-30T06:42:28.442ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34032025-04-30T06:42:28.444ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34042025-04-30T06:42:28.451ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34052025-04-30T06:42:28.451ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34062025-04-30T06:42:28.451ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
34072025-04-30T06:42:28.452ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34082025-04-30T06:42:28.464ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
34092025-04-30T06:42:28.465ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
34102025-04-30T06:42:28.465ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 590.780432ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
34112025-04-30T06:42:28.465ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
34122025-04-30T06:42:28.465ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34132025-04-30T06:42:28.465ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34142025-04-30T06:42:28.465ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34152025-04-30T06:42:28.465ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34162025-04-30T06:42:28.466ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34172025-04-30T06:42:28.466ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34182025-04-30T06:42:28.466ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
34192025-04-30T06:42:28.467ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34202025-04-30T06:42:28.469ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34212025-04-30T06:42:28.474ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34222025-04-30T06:42:28.474ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34232025-04-30T06:42:28.474ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34242025-04-30T06:42:28.475ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34252025-04-30T06:42:28.479ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
34262025-04-30T06:42:28.479ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
34272025-04-30T06:42:28.480ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 53 af 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 S............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58344 req_id = 1ee0de63-10f7-41b4-a431-8dc360565e5b
34282025-04-30T06:42:28.480ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21423,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58344 req_id = 1ee0de63-10f7-41b4-a431-8dc360565e5b
34292025-04-30T06:42:28.481ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
34302025-04-30T06:42:28.481ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34312025-04-30T06:42:28.481ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58344 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1ee0de63-10f7-41b4-a431-8dc360565e5b
34322025-04-30T06:42:28.482ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4213, tv_nsec: 280900866 } })
34332025-04-30T06:42:28.482ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
34342025-04-30T06:42:28.482ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ca 00 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59453 req_id = e1e2ba69-ec05-4172-82f0-f627ad98e5d5
34352025-04-30T06:42:28.482ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51712,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59453 req_id = e1e2ba69-ec05-4172-82f0-f627ad98e5d5
34362025-04-30T06:42:28.482ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34372025-04-30T06:42:28.482ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59453 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e1e2ba69-ec05-4172-82f0-f627ad98e5d5
34382025-04-30T06:42:28.532ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
34392025-04-30T06:42:28.533ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d8 4b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .K..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59754 req_id = ce52fca3-a346-49b6-949f-c1fc812f329d
34402025-04-30T06:42:28.533ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55371,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59754 req_id = ce52fca3-a346-49b6-949f-c1fc812f329d
34412025-04-30T06:42:28.533ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
34422025-04-30T06:42:28.533ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55371, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59754 req_id = ce52fca3-a346-49b6-949f-c1fc812f329d
34432025-04-30T06:42:28.534ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
34442025-04-30T06:42:28.534ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
34452025-04-30T06:42:28.534ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
34462025-04-30T06:42:28.544ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "67e946ef-e655-47d3-94b5-f7185a2e3e3d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34472025-04-30T06:42:28.545ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
34482025-04-30T06:42:28.548ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "edbd97a0-c807-461e-8350-1ec57e138a12", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34492025-04-30T06:42:28.550ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
34502025-04-30T06:42:28.571ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2b693dd-843d-4f25-b481-0b507f479337", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34512025-04-30T06:42:28.571ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
34522025-04-30T06:42:28.575ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bcf1d3a-32cd-4320-ade5-bca71bda51d2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34532025-04-30T06:42:28.575ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
34542025-04-30T06:42:28.578ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58dee6e9-c198-4810-911d-b422d4c66410", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34552025-04-30T06:42:28.578ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
34562025-04-30T06:42:28.581ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c92f7542-195b-4c3c-a87a-feaac95afebe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34572025-04-30T06:42:28.581ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
34582025-04-30T06:42:28.592ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
34592025-04-30T06:42:28.592ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32a394f2-8c62-4707-b784-df8271f7a679", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34602025-04-30T06:42:28.593ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
34612025-04-30T06:42:28.597ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd0fbf46-f2f4-46e2-88c9-4fe27a0d9069", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34622025-04-30T06:42:28.597ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
34632025-04-30T06:42:28.600ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b72882e4-0d6e-4bae-b03f-8b2147d73532", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34642025-04-30T06:42:28.600ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
34652025-04-30T06:42:28.614ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66b1db15-2368-4e3d-93c7-f5cfc56cfab5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34662025-04-30T06:42:28.616ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
34672025-04-30T06:42:28.642ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b00d4647-ea40-48fb-8b71-5f3b68876585", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34682025-04-30T06:42:28.642ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
34692025-04-30T06:42:28.643ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34702025-04-30T06:42:28.643ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34712025-04-30T06:42:28.644ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34722025-04-30T06:42:28.644ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
34732025-04-30T06:42:28.645ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34742025-04-30T06:42:28.647ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66bb20cc-02dc-4c8a-bee8-5e56b0d3a201", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34752025-04-30T06:42:28.647ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
34762025-04-30T06:42:28.650ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34772025-04-30T06:42:28.685ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34782025-04-30T06:42:28.685ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34792025-04-30T06:42:28.685ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
34802025-04-30T06:42:28.686ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34812025-04-30T06:42:28.686ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1100cb19-b654-425e-a2b5-6fb8cdfcd8e2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34822025-04-30T06:42:28.686ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
34832025-04-30T06:42:28.690ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34842025-04-30T06:42:28.690ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
34852025-04-30T06:42:28.695ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34862025-04-30T06:42:28.701ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d4bbbc7-1238-42a2-a638-b461be0b024d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34872025-04-30T06:42:28.701ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
34882025-04-30T06:42:28.706ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56d2bf63-786d-4eac-a3ce-cc624f84ccdc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34892025-04-30T06:42:28.706ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
34902025-04-30T06:42:28.712ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f07b019-af17-4d19-ae51-832ca08ab522", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34912025-04-30T06:42:28.713ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
34922025-04-30T06:42:28.722ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b179e8c5-737a-4c2c-94bb-567462ddc161", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34932025-04-30T06:42:28.722ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
34942025-04-30T06:42:28.726ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56691ded-110d-4e35-9345-f3911dc17bc7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34952025-04-30T06:42:28.726ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
34962025-04-30T06:42:28.732ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9a40dcd-6d9a-47cd-9c17-a34dc0bfe10f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
34972025-04-30T06:42:28.732ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
34982025-04-30T06:42:28.732ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
34992025-04-30T06:42:28.734ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "409cdada-9022-41b9-8b4d-cb3fe198e36f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35002025-04-30T06:42:28.735ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
35012025-04-30T06:42:28.743ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de14f137-9402-411c-b27b-65aed7f57bab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35022025-04-30T06:42:28.743ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
35032025-04-30T06:42:28.745ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0e9262a-81d0-4972-a066-39dbf2b03bd2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35042025-04-30T06:42:28.746ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
35052025-04-30T06:42:28.747ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "616a773a-2e15-4210-935a-adc19746d7d8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35062025-04-30T06:42:28.747ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
35072025-04-30T06:42:28.749ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "564cbb17-82d5-4c4f-baea-ca502de3d534", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35082025-04-30T06:42:28.750ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
35092025-04-30T06:42:28.752ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62988b03-8094-496f-9aaf-dd03988a6b32", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35102025-04-30T06:42:28.752ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
35112025-04-30T06:42:28.755ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd9bcd3c-9ca9-4add-90aa-ccf52799faae", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35122025-04-30T06:42:28.755ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
35132025-04-30T06:42:28.757ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42959dc3-74db-4484-8009-e36dab2a36c7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35142025-04-30T06:42:28.758ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
35152025-04-30T06:42:28.760ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "133fd525-6b7c-47a3-8bf7-0be6f25f4db3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35162025-04-30T06:42:28.760ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
35172025-04-30T06:42:28.763ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84add4b7-c5e0-4fc6-b828-13b61fcd1871", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35182025-04-30T06:42:28.763ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
35192025-04-30T06:42:28.765ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e72664af-2b17-4f70-8886-5c564fe93ba6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35202025-04-30T06:42:28.765ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
35212025-04-30T06:42:28.766ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27cd32c9-e410-47f0-bf7c-819f96785962", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35222025-04-30T06:42:28.766ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
35232025-04-30T06:42:28.767ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a7789ac-8818-4799-b456-c9dc817c126c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35242025-04-30T06:42:28.768ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
35252025-04-30T06:42:28.769ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a41819df-d1c7-4c7a-8ea7-72b061f74a65", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35262025-04-30T06:42:28.769ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
35272025-04-30T06:42:28.770ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9150a5c-0a5f-4edd-83c2-6c3ebe96cc3b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35282025-04-30T06:42:28.771ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
35292025-04-30T06:42:28.773ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fbd2d8c-b366-4e08-a846-d5966a09c199", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35302025-04-30T06:42:28.773ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
35312025-04-30T06:42:28.775ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bb9cfd8-df23-4bb1-9e4b-69f91fd6e9c8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35322025-04-30T06:42:28.776ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
35332025-04-30T06:42:28.779ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6fbd8f1-5b06-4a17-b39a-9c58b5b30fb0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35342025-04-30T06:42:28.779ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
35352025-04-30T06:42:28.784ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35362025-04-30T06:42:28.784ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35372025-04-30T06:42:28.784ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35382025-04-30T06:42:28.784ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
35392025-04-30T06:42:28.785ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35402025-04-30T06:42:28.787ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95c21617-285b-4955-ad6e-b1213b26b54c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35412025-04-30T06:42:28.788ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
35422025-04-30T06:42:28.788ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35432025-04-30T06:42:28.795ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35442025-04-30T06:42:28.795ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35452025-04-30T06:42:28.795ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
35462025-04-30T06:42:28.797ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35472025-04-30T06:42:28.802ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35482025-04-30T06:42:28.802ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
35492025-04-30T06:42:28.802ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35502025-04-30T06:42:28.804ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56b69789-5d16-4be5-8054-6bd7f9da122c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35512025-04-30T06:42:28.804ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
35522025-04-30T06:42:28.810ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33de6e9a-2275-4faa-adff-efc95fd076c5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35532025-04-30T06:42:28.811ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
35542025-04-30T06:42:28.815ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54a51d85-b3b1-493f-9885-17782596ca35", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35552025-04-30T06:42:28.815ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
35562025-04-30T06:42:28.817ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a37cd5a7-d4b5-43c8-8738-be1a39739bb3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35572025-04-30T06:42:28.818ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
35582025-04-30T06:42:28.821ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6464340-e6ac-468a-b6f8-218b5e013b00", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35592025-04-30T06:42:28.822ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
35602025-04-30T06:42:28.824ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c04e5cf8-5172-4e6c-bc47-c918ce31b83b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35612025-04-30T06:42:28.824ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
35622025-04-30T06:42:28.828ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09767e81-1a38-4790-8eb0-d11881716eab", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
35632025-04-30T06:42:28.828ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35642025-04-30T06:42:28.828ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35652025-04-30T06:42:28.828ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35662025-04-30T06:42:28.828ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35672025-04-30T06:42:28.828ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35682025-04-30T06:42:28.832ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35692025-04-30T06:42:28.836ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35702025-04-30T06:42:28.836ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35712025-04-30T06:42:28.836ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35722025-04-30T06:42:28.836ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35732025-04-30T06:42:28.836ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35742025-04-30T06:42:28.836ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35752025-04-30T06:42:28.841ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
35762025-04-30T06:42:28.846ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35772025-04-30T06:42:28.852ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35782025-04-30T06:42:28.852ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35792025-04-30T06:42:28.861ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35802025-04-30T06:42:28.862ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35812025-04-30T06:42:28.938ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
35822025-04-30T06:42:28.938ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
35832025-04-30T06:42:28.939ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35842025-04-30T06:42:28.939ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35852025-04-30T06:42:28.939ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35862025-04-30T06:42:28.939ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
35872025-04-30T06:42:28.941ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35882025-04-30T06:42:28.941ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:47631 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
35892025-04-30T06:42:28.942ZTRACtest_delete_snapshot (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35902025-04-30T06:42:28.942ZTRACtest_delete_snapshot (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35912025-04-30T06:42:28.942ZDEBGtest_delete_snapshot (producer-registration-task): will register with Nexus at 127.0.0.1:35404
35922025-04-30T06:42:29.023ZDEBGtest_delete_snapshot (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:35404/metrics/producers
35932025-04-30T06:42:29.027ZINFOtest_delete_snapshot (dropshot_internal): accepted connection local_addr = 127.0.0.1:35404 remote_addr = 127.0.0.1:63895
35942025-04-30T06:42:29.027ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:47631 req_id = 0c59b8f7-2360-4392-8f9a-389bce2f0499 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35952025-04-30T06:42:29.028ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1123 local_addr = [::1]:62607 method = GET remote_addr = [::1]:47631 req_id = 0c59b8f7-2360-4392-8f9a-389bce2f0499 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35962025-04-30T06:42:29.028ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35972025-04-30T06:42:29.039ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35982025-04-30T06:42:29.039ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35992025-04-30T06:42:29.039ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
36002025-04-30T06:42:29.039ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36012025-04-30T06:42:29.040ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:63895 req_id = 8b46a194-0b23-4284-9f39-255dea47517f uri = /metrics/producers
36022025-04-30T06:42:29.040ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c59b8f7-2360-4392-8f9a-389bce2f0499", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36032025-04-30T06:42:29.044ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36042025-04-30T06:42:29.044ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36052025-04-30T06:42:29.044ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
36062025-04-30T06:42:29.046ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36072025-04-30T06:42:29.046ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36082025-04-30T06:42:29.046ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36092025-04-30T06:42:29.046ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36102025-04-30T06:42:29.047ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36112025-04-30T06:42:29.047ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:63895 req_id = 8b46a194-0b23-4284-9f39-255dea47517f resource = Database uri = /metrics/producers
36122025-04-30T06:42:29.047ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:63895 req_id = 8b46a194-0b23-4284-9f39-255dea47517f roles = RoleSet { roles: {} } uri = /metrics/producers
36132025-04-30T06:42:29.048ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:63895 req_id = 8b46a194-0b23-4284-9f39-255dea47517f resource = Database result = Ok(()) uri = /metrics/producers
36142025-04-30T06:42:29.052ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36152025-04-30T06:42:29.052ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
36162025-04-30T06:42:29.056ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36172025-04-30T06:42:29.056ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36182025-04-30T06:42:29.064ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36192025-04-30T06:42:29.064ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36202025-04-30T06:42:29.064ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36212025-04-30T06:42:29.065ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36222025-04-30T06:42:29.073ZINFOtest_delete_snapshot (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
36232025-04-30T06:42:29.073ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 33889 local_addr = 127.0.0.1:35404 method = POST remote_addr = 127.0.0.1:63895 req_id = 8b46a194-0b23-4284-9f39-255dea47517f response_code = 201 uri = /metrics/producers
36242025-04-30T06:42:29.074ZDEBGtest_delete_snapshot (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "8b46a194-0b23-4284-9f39-255dea47517f", "content-length": "41", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36252025-04-30T06:42:29.076ZDEBGtest_delete_snapshot (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
36262025-04-30T06:42:29.166ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
36272025-04-30T06:42:29.202ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
36282025-04-30T06:42:29.203ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36292025-04-30T06:42:29.203ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 737.284131ms iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36302025-04-30T06:42:29.203ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
36312025-04-30T06:42:29.203ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36322025-04-30T06:42:29.203ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36332025-04-30T06:42:29.203ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36342025-04-30T06:42:29.203ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36352025-04-30T06:42:29.203ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36362025-04-30T06:42:29.206ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36372025-04-30T06:42:29.211ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36382025-04-30T06:42:29.211ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36392025-04-30T06:42:29.211ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36402025-04-30T06:42:29.212ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36412025-04-30T06:42:29.216ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
36422025-04-30T06:42:29.216ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
36432025-04-30T06:42:29.217ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36442025-04-30T06:42:29.217ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36452025-04-30T06:42:29.217ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36462025-04-30T06:42:29.217ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
36472025-04-30T06:42:29.218ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36482025-04-30T06:42:29.226ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 03 76 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .v..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59510 req_id = 69375d17-9f9a-4b9c-9696-fcda16810745
36492025-04-30T06:42:29.229ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 886,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59510 req_id = 69375d17-9f9a-4b9c-9696-fcda16810745
36502025-04-30T06:42:29.229ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
36512025-04-30T06:42:29.229ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36522025-04-30T06:42:29.229ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59510 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 69375d17-9f9a-4b9c-9696-fcda16810745
36532025-04-30T06:42:29.230ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4214, tv_nsec: 29593247 } })
36542025-04-30T06:42:29.230ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
36552025-04-30T06:42:29.231ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1c 8f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50394 req_id = 73610ba6-e853-452f-a38c-8102e7473e02
36562025-04-30T06:42:29.231ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7311,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50394 req_id = 73610ba6-e853-452f-a38c-8102e7473e02
36572025-04-30T06:42:29.231ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36582025-04-30T06:42:29.231ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50394 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 73610ba6-e853-452f-a38c-8102e7473e02
36592025-04-30T06:42:29.231ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36602025-04-30T06:42:29.237ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36612025-04-30T06:42:29.238ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36622025-04-30T06:42:29.238ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
36632025-04-30T06:42:29.238ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36642025-04-30T06:42:29.300ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
36652025-04-30T06:42:29.301ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6d fd 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c m............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62826 req_id = 95f9c844-d643-4d67-9eb4-38648b247d64
36662025-04-30T06:42:29.301ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28157,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62826 req_id = 95f9c844-d643-4d67-9eb4-38648b247d64
36672025-04-30T06:42:29.301ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
36682025-04-30T06:42:29.301ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28157, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62826 req_id = 95f9c844-d643-4d67-9eb4-38648b247d64
36692025-04-30T06:42:29.301ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
36702025-04-30T06:42:29.301ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
36712025-04-30T06:42:29.301ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
36722025-04-30T06:42:29.303ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36732025-04-30T06:42:29.303ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
36742025-04-30T06:42:29.304ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36752025-04-30T06:42:29.304ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3df5b17a-2cbe-4c00-981d-a6fab6292ba5", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36762025-04-30T06:42:29.305ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
36772025-04-30T06:42:29.307ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49f60c6a-5287-46ab-9aad-988a2e4a8231", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36782025-04-30T06:42:29.307ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
36792025-04-30T06:42:29.309ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e893bf6f-55ab-41e7-8745-560965b7dd13", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36802025-04-30T06:42:29.309ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
36812025-04-30T06:42:29.311ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "801e53e5-f39b-4f35-88fc-37834bd53021", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36822025-04-30T06:42:29.311ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
36832025-04-30T06:42:29.314ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78722ad9-1e80-4d5c-8a91-92e6179028a7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36842025-04-30T06:42:29.314ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
36852025-04-30T06:42:29.316ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28fdc320-073c-42e4-ad3b-5b9eea7e809f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36862025-04-30T06:42:29.317ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
36872025-04-30T06:42:29.319ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34c9765f-dfb4-4a73-9cae-f124ca8a9299", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36882025-04-30T06:42:29.319ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
36892025-04-30T06:42:29.321ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16020f5c-1179-424c-8096-cfc7ba0fbeec", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36902025-04-30T06:42:29.321ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
36912025-04-30T06:42:29.323ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49ed9b54-9b7a-4554-bc84-b3ae403c65cc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36922025-04-30T06:42:29.323ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
36932025-04-30T06:42:29.325ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f4f86c2-8380-4205-acec-17b7ba2d1f55", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36942025-04-30T06:42:29.325ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
36952025-04-30T06:42:29.326ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fa59b0f-7cab-4e95-b7d6-5625390e266a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36962025-04-30T06:42:29.326ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
36972025-04-30T06:42:29.327ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f68107b6-1d08-4b64-bee6-8ce1133ad9bc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
36982025-04-30T06:42:29.327ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
36992025-04-30T06:42:29.332ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b392733-2491-4698-a304-ac6d2ec98ec1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37002025-04-30T06:42:29.332ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
37012025-04-30T06:42:29.335ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "010b56e1-e9e1-4c25-900d-c038a0246683", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37022025-04-30T06:42:29.335ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
37032025-04-30T06:42:29.338ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7903af97-f2b9-444e-9721-7cc236366f63", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37042025-04-30T06:42:29.338ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
37052025-04-30T06:42:29.342ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
37062025-04-30T06:42:29.345ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ded6b302-e017-4772-8de9-531860c46c6e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37072025-04-30T06:42:29.345ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
37082025-04-30T06:42:29.346ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b34108d4-7e21-455f-ad63-a7bee83a522a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37092025-04-30T06:42:29.346ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
37102025-04-30T06:42:29.348ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95a58bca-a622-4a10-8cc6-f822aa41455a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37112025-04-30T06:42:29.349ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
37122025-04-30T06:42:29.352ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a7eff04-f81a-4924-bdbe-569f2f6bace5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37132025-04-30T06:42:29.352ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
37142025-04-30T06:42:29.353ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e92bfae-0a84-4b5b-967c-51d436a4847e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37152025-04-30T06:42:29.353ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
37162025-04-30T06:42:29.354ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a1d53c3-f955-4cb2-9705-17e64b87dc16", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37172025-04-30T06:42:29.354ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
37182025-04-30T06:42:29.355ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4c0820c-e7b4-4520-b972-188f305ec950", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37192025-04-30T06:42:29.355ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
37202025-04-30T06:42:29.357ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32f3169c-a630-40ca-9a97-737b433680f4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37212025-04-30T06:42:29.357ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
37222025-04-30T06:42:29.358ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b00f16ef-6632-4790-9d37-9c9e0da121b0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37232025-04-30T06:42:29.358ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
37242025-04-30T06:42:29.365ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c06eb4e8-3d4b-49e7-bb2f-a3718d1a18df", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37252025-04-30T06:42:29.365ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
37262025-04-30T06:42:29.370ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "222d3516-9065-4961-ae7a-28b16981bd8f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37272025-04-30T06:42:29.370ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
37282025-04-30T06:42:29.372ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e8df670-0db1-4d4d-85ee-937ea5ad9592", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37292025-04-30T06:42:29.372ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
37302025-04-30T06:42:29.377ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "081d1b8f-b5c5-49cd-bf06-b739b7852b96", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37312025-04-30T06:42:29.377ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
37322025-04-30T06:42:29.381ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "331f891a-e618-498a-a781-ac3ac31eb926", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37332025-04-30T06:42:29.381ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
37342025-04-30T06:42:29.386ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1727075-1713-459b-9240-dbd4d4027be4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37352025-04-30T06:42:29.386ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
37362025-04-30T06:42:29.388ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "030a3de4-ac22-43a7-b06a-50d9fb126ba0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37372025-04-30T06:42:29.388ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
37382025-04-30T06:42:29.389ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdd21068-bf48-4ecf-9aa0-7d5739c63b71", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37392025-04-30T06:42:29.389ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
37402025-04-30T06:42:29.391ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "896c0a94-10d9-4af8-9d0a-bf66afb8c683", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37412025-04-30T06:42:29.391ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
37422025-04-30T06:42:29.392ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99ed82aa-fbf2-4d9e-adf2-c4024c7b52ad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37432025-04-30T06:42:29.392ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
37442025-04-30T06:42:29.394ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37452025-04-30T06:42:29.394ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37462025-04-30T06:42:29.394ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37472025-04-30T06:42:29.394ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
37482025-04-30T06:42:29.395ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37492025-04-30T06:42:29.397ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bcd6a16-8a1e-4803-a83e-c640ae98aed0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37502025-04-30T06:42:29.397ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
37512025-04-30T06:42:29.399ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37522025-04-30T06:42:29.413ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37532025-04-30T06:42:29.413ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37542025-04-30T06:42:29.413ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
37552025-04-30T06:42:29.414ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37562025-04-30T06:42:29.416ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37572025-04-30T06:42:29.416ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
37582025-04-30T06:42:29.418ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37592025-04-30T06:42:29.419ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ba36d1d-c762-40d4-9ae6-3a176e2d6580", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37602025-04-30T06:42:29.419ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
37612025-04-30T06:42:29.424ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9de0d724-e785-49aa-b700-391675ac014d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37622025-04-30T06:42:29.424ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
37632025-04-30T06:42:29.427ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dccd533e-0d9d-4a1e-8642-0f527a939cf2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37642025-04-30T06:42:29.427ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
37652025-04-30T06:42:29.431ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbbd83d7-6fde-48db-ac7d-8bf1665424e2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37662025-04-30T06:42:29.431ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
37672025-04-30T06:42:29.455ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29655c86-60e0-4b12-8b6c-33b7a3aac2f0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37682025-04-30T06:42:29.457ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
37692025-04-30T06:42:29.461ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76bba180-9b7d-4cb7-be2c-596508dbb4e7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37702025-04-30T06:42:29.462ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
37712025-04-30T06:42:29.464ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b50cdf0b-1ee8-4b68-9ed6-a73314c840df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37722025-04-30T06:42:29.464ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
37732025-04-30T06:42:29.466ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d824dee-20ec-4b98-975a-05af7391d23c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37742025-04-30T06:42:29.466ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
37752025-04-30T06:42:29.471ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6471ccb2-d1c9-43f9-a11d-4b1c77d15057", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37762025-04-30T06:42:29.471ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
37772025-04-30T06:42:29.472ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
37782025-04-30T06:42:29.482ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7880758-13a7-4c70-b991-c0cd6c316e0c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
37792025-04-30T06:42:29.482ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37802025-04-30T06:42:29.482ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37812025-04-30T06:42:29.482ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37822025-04-30T06:42:29.482ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37832025-04-30T06:42:29.483ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37842025-04-30T06:42:29.485ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37852025-04-30T06:42:29.489ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37862025-04-30T06:42:29.489ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37872025-04-30T06:42:29.489ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37882025-04-30T06:42:29.490ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37892025-04-30T06:42:29.490ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37902025-04-30T06:42:29.490ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37912025-04-30T06:42:29.490ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
37922025-04-30T06:42:29.490ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37932025-04-30T06:42:29.491ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1089aa2b-8e14-457a-bf92-039093a5acfd uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37942025-04-30T06:42:29.491ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1089aa2b-8e14-457a-bf92-039093a5acfd resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37952025-04-30T06:42:29.491ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1089aa2b-8e14-457a-bf92-039093a5acfd roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37962025-04-30T06:42:29.492ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1089aa2b-8e14-457a-bf92-039093a5acfd resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37972025-04-30T06:42:29.492ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37982025-04-30T06:42:29.498ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37992025-04-30T06:42:29.498ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38002025-04-30T06:42:29.498ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38012025-04-30T06:42:29.498ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38022025-04-30T06:42:29.500ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 9404 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1089aa2b-8e14-457a-bf92-039093a5acfd response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38032025-04-30T06:42:29.501ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "1089aa2b-8e14-457a-bf92-039093a5acfd", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:28 GMT"} })
38042025-04-30T06:42:29.501ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
38052025-04-30T06:42:29.501ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8b8aafcb-4418-4bbc-a5dd-410c46dcd197 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
38062025-04-30T06:42:29.502ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8b8aafcb-4418-4bbc-a5dd-410c46dcd197 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
38072025-04-30T06:42:29.502ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8b8aafcb-4418-4bbc-a5dd-410c46dcd197 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
38082025-04-30T06:42:29.502ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8b8aafcb-4418-4bbc-a5dd-410c46dcd197 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
38092025-04-30T06:42:29.547ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
38102025-04-30T06:42:29.547ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
38112025-04-30T06:42:29.547ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38122025-04-30T06:42:29.547ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38132025-04-30T06:42:29.547ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38142025-04-30T06:42:29.547ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
38152025-04-30T06:42:29.548ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38162025-04-30T06:42:29.548ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:41570 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
38172025-04-30T06:42:29.549ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:41570 req_id = c2b3c668-dd24-4f99-86f2-e3049c9f7b86 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38182025-04-30T06:42:29.549ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 443 local_addr = [::1]:62607 method = GET remote_addr = [::1]:41570 req_id = c2b3c668-dd24-4f99-86f2-e3049c9f7b86 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38192025-04-30T06:42:29.550ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 48543 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8b8aafcb-4418-4bbc-a5dd-410c46dcd197 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
38202025-04-30T06:42:29.551ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b8aafcb-4418-4bbc-a5dd-410c46dcd197", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
38212025-04-30T06:42:29.551ZDEBGtest_delete_snapshot (oximeter-agent): registered new metric producer address = [::1]:44480 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
38222025-04-30T06:42:29.552ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2b3c668-dd24-4f99-86f2-e3049c9f7b86", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
38232025-04-30T06:42:29.555ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38242025-04-30T06:42:29.555ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38252025-04-30T06:42:29.555ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
38262025-04-30T06:42:29.560ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38272025-04-30T06:42:29.560ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38282025-04-30T06:42:29.560ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38292025-04-30T06:42:29.560ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38302025-04-30T06:42:29.561ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38312025-04-30T06:42:29.615ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
38322025-04-30T06:42:29.616ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38332025-04-30T06:42:29.621ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38342025-04-30T06:42:29.622ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38352025-04-30T06:42:29.622ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
38362025-04-30T06:42:29.622ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38372025-04-30T06:42:29.626ZDEBGtest_delete_snapshot (oximeter-agent): registered new metric producer address = 127.0.0.1:65039 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
38382025-04-30T06:42:29.743ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
38392025-04-30T06:42:29.743ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38402025-04-30T06:42:29.751ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38412025-04-30T06:42:29.751ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38422025-04-30T06:42:29.751ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38432025-04-30T06:42:29.752ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38442025-04-30T06:42:29.755ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38452025-04-30T06:42:29.755ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
38462025-04-30T06:42:29.755ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38472025-04-30T06:42:29.756ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
38482025-04-30T06:42:29.829ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
38492025-04-30T06:42:29.831ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
38502025-04-30T06:42:29.832ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 629.566661ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
38512025-04-30T06:42:29.832ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
38522025-04-30T06:42:29.832ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38532025-04-30T06:42:29.833ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38542025-04-30T06:42:29.833ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38552025-04-30T06:42:29.833ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38562025-04-30T06:42:29.834ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38572025-04-30T06:42:29.835ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
38582025-04-30T06:42:29.838ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38592025-04-30T06:42:29.845ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38602025-04-30T06:42:29.846ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38612025-04-30T06:42:29.846ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38622025-04-30T06:42:29.847ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38632025-04-30T06:42:29.856ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
38642025-04-30T06:42:29.856ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
38652025-04-30T06:42:29.858ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 96 e1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51533 req_id = bdf4f2c6-27f9-40e2-b9b5-5811c743729b
38662025-04-30T06:42:29.858ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38625,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51533 req_id = bdf4f2c6-27f9-40e2-b9b5-5811c743729b
38672025-04-30T06:42:29.858ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
38682025-04-30T06:42:29.858ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38692025-04-30T06:42:29.858ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51533 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bdf4f2c6-27f9-40e2-b9b5-5811c743729b
38702025-04-30T06:42:29.860ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4214, tv_nsec: 658863928 } })
38712025-04-30T06:42:29.860ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
38722025-04-30T06:42:29.860ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 50 3f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e P?..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58330 req_id = 3110d9b9-1e17-418b-ba3d-d423c7b40c27
38732025-04-30T06:42:29.860ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20543,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58330 req_id = 3110d9b9-1e17-418b-ba3d-d423c7b40c27
38742025-04-30T06:42:29.860ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38752025-04-30T06:42:29.861ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58330 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3110d9b9-1e17-418b-ba3d-d423c7b40c27
38762025-04-30T06:42:29.916ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
38772025-04-30T06:42:29.916ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38782025-04-30T06:42:29.916ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38792025-04-30T06:42:29.916ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38802025-04-30T06:42:29.916ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
38812025-04-30T06:42:29.917ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38822025-04-30T06:42:29.917ZTRACtest_delete_snapshot (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38832025-04-30T06:42:29.917ZTRACtest_delete_snapshot (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38842025-04-30T06:42:29.918ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: be e4 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53143 req_id = c8c54b8d-f364-498c-88da-93dd2895738e
38852025-04-30T06:42:29.918ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48868,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53143 req_id = c8c54b8d-f364-498c-88da-93dd2895738e
38862025-04-30T06:42:29.918ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
38872025-04-30T06:42:29.918ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48868, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53143 req_id = c8c54b8d-f364-498c-88da-93dd2895738e
38882025-04-30T06:42:29.918ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
38892025-04-30T06:42:29.918ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
38902025-04-30T06:42:29.918ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
38912025-04-30T06:42:29.920ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38922025-04-30T06:42:29.928ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38932025-04-30T06:42:29.931ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38942025-04-30T06:42:29.931ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
38952025-04-30T06:42:29.931ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38962025-04-30T06:42:29.933ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "62f98a58-5894-4675-aa8e-b4b98275aecf", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
38972025-04-30T06:42:29.934ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
38982025-04-30T06:42:29.935ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38992025-04-30T06:42:29.935ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
39002025-04-30T06:42:29.935ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39012025-04-30T06:42:29.936ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c158726-481a-4607-8e4b-4bb41e4ae071", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39022025-04-30T06:42:29.936ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
39032025-04-30T06:42:29.938ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58fe0d76-ce4f-4dde-87b2-7b6d83ab5b2b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39042025-04-30T06:42:29.938ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
39052025-04-30T06:42:29.941ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7f15d78-bfb3-4857-95af-dd90b5c45f16", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39062025-04-30T06:42:29.942ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
39072025-04-30T06:42:29.944ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afa54540-91bb-4af2-a069-5d4e405ce3e8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39082025-04-30T06:42:29.944ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
39092025-04-30T06:42:29.948ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d3d58fc-eb28-4475-9e60-c93e2e383eb9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39102025-04-30T06:42:29.948ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
39112025-04-30T06:42:29.952ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3a57520-b568-437b-9451-f185086568f0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39122025-04-30T06:42:29.952ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
39132025-04-30T06:42:29.954ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "502594ed-0992-41da-b330-6e3a30c36666", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39142025-04-30T06:42:29.954ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
39152025-04-30T06:42:29.956ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33150437-5ef9-4814-beca-985fb664fa9e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39162025-04-30T06:42:29.956ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
39172025-04-30T06:42:29.957ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbeb3b2d-2d1e-40b2-9f7a-516367f90758", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39182025-04-30T06:42:29.957ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
39192025-04-30T06:42:29.958ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b28b12a-201e-4335-bae9-661d1a2ae246", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39202025-04-30T06:42:29.959ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
39212025-04-30T06:42:29.960ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "326e46f4-f6df-4a14-8c72-fbe2a5e4d20b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39222025-04-30T06:42:29.961ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
39232025-04-30T06:42:29.963ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "888289a7-6f4f-4c02-bf1b-cfdcec08d21e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39242025-04-30T06:42:29.964ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
39252025-04-30T06:42:29.968ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93d72608-eaa8-43b1-81c8-2e42d6f65568", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39262025-04-30T06:42:29.969ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
39272025-04-30T06:42:29.975ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b2f4ffa-2919-45b4-ba12-52b1140f2a4f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39282025-04-30T06:42:29.975ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
39292025-04-30T06:42:29.979ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3839892-8b85-45a5-8654-f441de32b45e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39302025-04-30T06:42:29.979ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
39312025-04-30T06:42:29.982ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac93134-770b-4f9c-a964-0df216367b28", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39322025-04-30T06:42:29.982ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
39332025-04-30T06:42:29.992ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d9a0a89-2392-4914-b07f-01e389fa016a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39342025-04-30T06:42:29.992ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
39352025-04-30T06:42:29.993ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
39362025-04-30T06:42:29.996ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a25805a2-f695-4872-bc8d-655a38a7bc64", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39372025-04-30T06:42:29.996ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
39382025-04-30T06:42:29.997ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a68b00bd-064b-4343-945e-cc13b6bf066a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39392025-04-30T06:42:29.997ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
39402025-04-30T06:42:30.029ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24cd1dd1-e4a0-4c4b-8338-9201a0e9ea4f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39412025-04-30T06:42:30.029ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
39422025-04-30T06:42:30.031ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bdd63f4-defc-4228-9f73-54cd5cafcb0a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39432025-04-30T06:42:30.031ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
39442025-04-30T06:42:30.033ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1db323f-f9c7-4553-89d6-3431066c764a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39452025-04-30T06:42:30.033ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
39462025-04-30T06:42:30.035ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ad173fe-63f6-4677-8481-b3826b411d9e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39472025-04-30T06:42:30.036ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
39482025-04-30T06:42:30.040ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25028d23-7d28-4f3a-96d5-e8e9677f1441", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39492025-04-30T06:42:30.040ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
39502025-04-30T06:42:30.044ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5f5550e-6009-4f36-a405-e4d9c5a47cef", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39512025-04-30T06:42:30.044ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
39522025-04-30T06:42:30.044ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39532025-04-30T06:42:30.044ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39542025-04-30T06:42:30.044ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39552025-04-30T06:42:30.044ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
39562025-04-30T06:42:30.045ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39572025-04-30T06:42:30.047ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07afb004-bf66-4109-8893-aec929a94bf6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39582025-04-30T06:42:30.047ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
39592025-04-30T06:42:30.047ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39602025-04-30T06:42:30.052ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39612025-04-30T06:42:30.052ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39622025-04-30T06:42:30.052ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
39632025-04-30T06:42:30.053ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39642025-04-30T06:42:30.055ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2e2ab0d-74c2-4fcd-ae4e-ec0ad1094615", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39652025-04-30T06:42:30.055ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
39662025-04-30T06:42:30.056ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39672025-04-30T06:42:30.056ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
39682025-04-30T06:42:30.056ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39692025-04-30T06:42:30.060ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "360bf446-1444-4b0f-96bd-0d7199a634a6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39702025-04-30T06:42:30.060ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
39712025-04-30T06:42:30.071ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "208b11c7-e503-40e7-b08c-890333023ee9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39722025-04-30T06:42:30.071ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
39732025-04-30T06:42:30.081ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e87515f-c93f-4fa5-a608-86ea6e1f2ec4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39742025-04-30T06:42:30.081ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
39752025-04-30T06:42:30.097ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "adc58b73-aefd-4b61-a580-9513d2c0107f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39762025-04-30T06:42:30.097ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
39772025-04-30T06:42:30.109ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6db3c98-63e1-4b0a-af34-fa50b6f75a47", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39782025-04-30T06:42:30.109ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
39792025-04-30T06:42:30.114ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0f2c68f-53b0-400a-9566-734c306677c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39802025-04-30T06:42:30.114ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
39812025-04-30T06:42:30.118ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f7dc0b7-3b0f-465f-b7e6-6fed8fb55bfc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39822025-04-30T06:42:30.119ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
39832025-04-30T06:42:30.119ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
39842025-04-30T06:42:30.123ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eda98af4-ba62-4748-a35a-b86ecd92627f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39852025-04-30T06:42:30.123ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
39862025-04-30T06:42:30.131ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fc8e3cf-fd0f-44ad-be2c-38cca7d63345", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39872025-04-30T06:42:30.131ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
39882025-04-30T06:42:30.134ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "317cbda8-f7fb-49d2-b778-f3ee749683c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39892025-04-30T06:42:30.134ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
39902025-04-30T06:42:30.136ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c9f00a1-4b5f-40cb-a064-7df62e08004f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39912025-04-30T06:42:30.144ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
39922025-04-30T06:42:30.147ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7073e7ba-ce30-45bd-8fb6-bd258bf48563", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39932025-04-30T06:42:30.148ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
39942025-04-30T06:42:30.150ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97fa6cec-904f-41f1-87a7-a4f7377b09c4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39952025-04-30T06:42:30.150ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
39962025-04-30T06:42:30.152ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f27513c-ab8e-4741-877d-e40eb421c66f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39972025-04-30T06:42:30.152ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
39982025-04-30T06:42:30.153ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "035c6c05-03ec-40da-b709-198728628d8d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
39992025-04-30T06:42:30.153ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
40002025-04-30T06:42:30.154ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ac92cef-6abc-4ec9-b7ab-43e63014469c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
40012025-04-30T06:42:30.154ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
40022025-04-30T06:42:30.155ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73189100-57f0-431f-b12c-6d4aa16834ee", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
40032025-04-30T06:42:30.155ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40042025-04-30T06:42:30.155ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40052025-04-30T06:42:30.155ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40062025-04-30T06:42:30.155ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40072025-04-30T06:42:30.158ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40082025-04-30T06:42:30.160ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40092025-04-30T06:42:30.164ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40102025-04-30T06:42:30.164ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40112025-04-30T06:42:30.164ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40122025-04-30T06:42:30.164ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40132025-04-30T06:42:30.164ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40142025-04-30T06:42:30.165ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40152025-04-30T06:42:30.170ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40162025-04-30T06:42:30.171ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40172025-04-30T06:42:30.171ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40182025-04-30T06:42:30.171ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
40192025-04-30T06:42:30.171ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40202025-04-30T06:42:30.171ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40212025-04-30T06:42:30.175ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40222025-04-30T06:42:30.175ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40232025-04-30T06:42:30.175ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40242025-04-30T06:42:30.175ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40252025-04-30T06:42:30.178ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40262025-04-30T06:42:30.184ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40272025-04-30T06:42:30.184ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40282025-04-30T06:42:30.184ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
40292025-04-30T06:42:30.185ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40302025-04-30T06:42:30.252ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
40312025-04-30T06:42:30.252ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
40322025-04-30T06:42:30.254ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40332025-04-30T06:42:30.254ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
40342025-04-30T06:42:30.255ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40352025-04-30T06:42:30.256ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:61691 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
40362025-04-30T06:42:30.257ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:61691 req_id = ec3c9224-69de-45b3-84a1-9d5ab65cbecc sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40372025-04-30T06:42:30.258ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 811 local_addr = [::1]:62607 method = GET remote_addr = [::1]:61691 req_id = ec3c9224-69de-45b3-84a1-9d5ab65cbecc response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40382025-04-30T06:42:30.260ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec3c9224-69de-45b3-84a1-9d5ab65cbecc", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
40392025-04-30T06:42:30.262ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40402025-04-30T06:42:30.262ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40412025-04-30T06:42:30.262ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
40422025-04-30T06:42:30.264ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40432025-04-30T06:42:30.264ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40442025-04-30T06:42:30.265ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40452025-04-30T06:42:30.265ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40462025-04-30T06:42:30.267ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40472025-04-30T06:42:30.272ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40482025-04-30T06:42:30.278ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40492025-04-30T06:42:30.278ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40502025-04-30T06:42:30.278ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40512025-04-30T06:42:30.279ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40522025-04-30T06:42:30.326ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
40532025-04-30T06:42:30.345ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
40542025-04-30T06:42:30.346ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
40552025-04-30T06:42:30.346ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 513.990903ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
40562025-04-30T06:42:30.346ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
40572025-04-30T06:42:30.346ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40582025-04-30T06:42:30.346ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40592025-04-30T06:42:30.346ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40602025-04-30T06:42:30.346ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40612025-04-30T06:42:30.347ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40622025-04-30T06:42:30.353ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40632025-04-30T06:42:30.357ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40642025-04-30T06:42:30.357ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40652025-04-30T06:42:30.357ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40662025-04-30T06:42:30.358ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40672025-04-30T06:42:30.361ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
40682025-04-30T06:42:30.361ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
40692025-04-30T06:42:30.362ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 13 5f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ._..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64874 req_id = 8698ffcc-e25b-47df-ad4f-3db49bc85e61
40702025-04-30T06:42:30.363ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4959,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64874 req_id = 8698ffcc-e25b-47df-ad4f-3db49bc85e61
40712025-04-30T06:42:30.363ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
40722025-04-30T06:42:30.363ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40732025-04-30T06:42:30.363ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64874 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8698ffcc-e25b-47df-ad4f-3db49bc85e61
40742025-04-30T06:42:30.364ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4215, tv_nsec: 163212660 } })
40752025-04-30T06:42:30.364ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
40762025-04-30T06:42:30.365ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 15 62 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .b..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52206 req_id = edaa757e-31f6-4747-b27b-728c5e51a683
40772025-04-30T06:42:30.365ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5474,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52206 req_id = edaa757e-31f6-4747-b27b-728c5e51a683
40782025-04-30T06:42:30.365ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40792025-04-30T06:42:30.365ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52206 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = edaa757e-31f6-4747-b27b-728c5e51a683
40802025-04-30T06:42:30.444ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
40812025-04-30T06:42:30.444ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40822025-04-30T06:42:30.444ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40832025-04-30T06:42:30.444ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40842025-04-30T06:42:30.444ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
40852025-04-30T06:42:30.445ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40862025-04-30T06:42:30.446ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 95 e5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63937 req_id = 25ddee16-1966-4402-bdff-82e26f069c95
40872025-04-30T06:42:30.446ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38373,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63937 req_id = 25ddee16-1966-4402-bdff-82e26f069c95
40882025-04-30T06:42:30.446ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
40892025-04-30T06:42:30.446ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38373, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63937 req_id = 25ddee16-1966-4402-bdff-82e26f069c95
40902025-04-30T06:42:30.447ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
40912025-04-30T06:42:30.447ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
40922025-04-30T06:42:30.447ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
40932025-04-30T06:42:30.448ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40942025-04-30T06:42:30.455ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40952025-04-30T06:42:30.455ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40962025-04-30T06:42:30.455ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
40972025-04-30T06:42:30.456ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40982025-04-30T06:42:30.461ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "117ac9f9-a707-447b-9c0b-530a3f9430e7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
40992025-04-30T06:42:30.461ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
41002025-04-30T06:42:30.461ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41012025-04-30T06:42:30.461ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
41022025-04-30T06:42:30.463ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41032025-04-30T06:42:30.465ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68bac570-3de6-4c8e-ab5e-99881eefa32c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41042025-04-30T06:42:30.465ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
41052025-04-30T06:42:30.468ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cd70021-93ee-45eb-87ef-7b51f4bb2876", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41062025-04-30T06:42:30.469ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
41072025-04-30T06:42:30.471ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87c01c49-e07e-4d37-b1d5-a1dd0a291869", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41082025-04-30T06:42:30.471ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
41092025-04-30T06:42:30.473ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d960a5db-5e5c-427b-a897-8f402585a265", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41102025-04-30T06:42:30.474ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
41112025-04-30T06:42:30.478ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93424bb2-e15e-4bc6-b32f-2bbd297d3803", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41122025-04-30T06:42:30.478ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
41132025-04-30T06:42:30.481ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc79b3b2-7fa2-4126-99cd-6a0e34e5f240", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41142025-04-30T06:42:30.481ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
41152025-04-30T06:42:30.484ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17f22516-2744-4624-8236-769a33748858", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41162025-04-30T06:42:30.484ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
41172025-04-30T06:42:30.485ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f72881b9-7f25-4424-ba0f-10791cf62274", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41182025-04-30T06:42:30.485ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
41192025-04-30T06:42:30.487ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d0c61a1-732f-4787-91ea-bc4c28e11ca9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41202025-04-30T06:42:30.487ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
41212025-04-30T06:42:30.488ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7ba1010-7734-4644-ad6f-55dc593f26f4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41222025-04-30T06:42:30.488ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
41232025-04-30T06:42:30.491ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
41242025-04-30T06:42:30.491ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c94be6ad-8590-4af8-9e87-5596c177bcfb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41252025-04-30T06:42:30.492ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
41262025-04-30T06:42:30.495ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca956fa4-10d7-4d9d-8aaf-83a7d10a0987", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41272025-04-30T06:42:30.495ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
41282025-04-30T06:42:30.498ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d148d49-559f-489e-b9e0-2cabe4fe65c9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41292025-04-30T06:42:30.499ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
41302025-04-30T06:42:30.503ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8c34078-465c-4dbc-ace8-dd8f079c6ef1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41312025-04-30T06:42:30.503ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
41322025-04-30T06:42:30.505ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d0acd11-8e3e-4acb-a58d-0d572c8e733d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41332025-04-30T06:42:30.505ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
41342025-04-30T06:42:30.507ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "702100b1-e6f1-4868-85af-b19c58bb1340", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41352025-04-30T06:42:30.507ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
41362025-04-30T06:42:30.510ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63936fff-26ab-4bad-8824-1e7e39d12e29", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41372025-04-30T06:42:30.511ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
41382025-04-30T06:42:30.514ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4602892c-4c2a-4fba-8366-3f622de802c4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41392025-04-30T06:42:30.514ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
41402025-04-30T06:42:30.516ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "018bd8f3-ec14-4565-830b-aef4b71624bd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41412025-04-30T06:42:30.516ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
41422025-04-30T06:42:30.520ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d3920e2-862c-46bc-9385-aef20c670de0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41432025-04-30T06:42:30.520ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
41442025-04-30T06:42:30.522ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64e6da0e-93c6-4834-9665-753856b93a6e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41452025-04-30T06:42:30.522ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
41462025-04-30T06:42:30.525ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eee2146-a4e6-490c-b8fd-6f73de24bee9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41472025-04-30T06:42:30.525ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
41482025-04-30T06:42:30.528ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f98b3d5-271c-41bf-b973-ee89cc432440", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41492025-04-30T06:42:30.528ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
41502025-04-30T06:42:30.531ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "576d1621-9673-41fa-86c7-7b682b2016e9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41512025-04-30T06:42:30.531ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
41522025-04-30T06:42:30.534ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f803f145-ec26-4d02-9b5c-6f8b204537c9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41532025-04-30T06:42:30.534ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
41542025-04-30T06:42:30.539ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94440564-5fbc-4d00-98d5-8e39dedfc116", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41552025-04-30T06:42:30.539ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
41562025-04-30T06:42:30.541ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a33c0064-d27d-4ae2-9555-a3a53144d737", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41572025-04-30T06:42:30.542ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
41582025-04-30T06:42:30.542ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41592025-04-30T06:42:30.542ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41602025-04-30T06:42:30.542ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41612025-04-30T06:42:30.542ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
41622025-04-30T06:42:30.543ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41632025-04-30T06:42:30.549ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2bd65f1-0ccc-4465-bb7c-7d1a6890332b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:29 GMT"} })
41642025-04-30T06:42:30.549ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
41652025-04-30T06:42:30.549ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41662025-04-30T06:42:30.557ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41672025-04-30T06:42:30.557ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41682025-04-30T06:42:30.557ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
41692025-04-30T06:42:30.558ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41702025-04-30T06:42:30.559ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41712025-04-30T06:42:30.560ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41722025-04-30T06:42:30.561ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41732025-04-30T06:42:30.561ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
41742025-04-30T06:42:30.563ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41752025-04-30T06:42:30.566ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41762025-04-30T06:42:30.568ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41772025-04-30T06:42:30.568ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41782025-04-30T06:42:30.568ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41792025-04-30T06:42:30.576ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96afb1e4-2a0b-40dd-8be1-54f483e8590b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41802025-04-30T06:42:30.577ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
41812025-04-30T06:42:30.579ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a516c1-5ef2-456e-9045-d5d777310e50", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41822025-04-30T06:42:30.579ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
41832025-04-30T06:42:30.581ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6ba47c1-c18f-4336-8623-0c5b5a7fc490", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41842025-04-30T06:42:30.581ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
41852025-04-30T06:42:30.582ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4985d433-1769-4f27-87ba-2ed0f89bbb74", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41862025-04-30T06:42:30.583ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
41872025-04-30T06:42:30.587ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "352960c6-73b7-4e03-a37e-5742c478429a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41882025-04-30T06:42:30.587ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
41892025-04-30T06:42:30.589ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7e6f6ff-f8c4-4511-b802-9a8495f28c8a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41902025-04-30T06:42:30.589ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
41912025-04-30T06:42:30.594ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85b59762-34e4-4f64-8125-48b107670a7c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41922025-04-30T06:42:30.594ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
41932025-04-30T06:42:30.597ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "689b29ba-1b0d-4827-8c59-fcbc4cc7c812", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41942025-04-30T06:42:30.598ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
41952025-04-30T06:42:30.601ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
41962025-04-30T06:42:30.603ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e8a7f77-f5dc-4959-a96b-4d9221dd209f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41972025-04-30T06:42:30.603ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
41982025-04-30T06:42:30.605ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1732501-a2a1-412b-a004-c374037d9a4d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
41992025-04-30T06:42:30.605ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
42002025-04-30T06:42:30.609ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78e58267-a413-44b0-9256-93e7c957f6ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
42012025-04-30T06:42:30.609ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
42022025-04-30T06:42:30.613ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc2a4fdd-a7ef-407d-870b-b4d789a453f1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
42032025-04-30T06:42:30.613ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
42042025-04-30T06:42:30.614ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e21eac4f-b7c5-46c2-b5cc-0d7b32bba542", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
42052025-04-30T06:42:30.615ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
42062025-04-30T06:42:30.616ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3b56507-dfbf-4dc7-9363-96eafd8768d4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
42072025-04-30T06:42:30.616ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
42082025-04-30T06:42:30.617ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b83470e8-d7bb-47e8-a7a5-06c257846ccd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
42092025-04-30T06:42:30.617ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
42102025-04-30T06:42:30.618ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "103ed748-5e4e-403d-9bf6-6618014ec024", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
42112025-04-30T06:42:30.619ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42122025-04-30T06:42:30.619ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42132025-04-30T06:42:30.619ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42142025-04-30T06:42:30.619ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42152025-04-30T06:42:30.620ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42162025-04-30T06:42:30.628ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42172025-04-30T06:42:30.633ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42182025-04-30T06:42:30.633ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42192025-04-30T06:42:30.633ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42202025-04-30T06:42:30.633ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42212025-04-30T06:42:30.634ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42222025-04-30T06:42:30.635ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42232025-04-30T06:42:30.638ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42242025-04-30T06:42:30.643ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42252025-04-30T06:42:30.643ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42262025-04-30T06:42:30.643ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42272025-04-30T06:42:30.643ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42282025-04-30T06:42:30.718ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
42292025-04-30T06:42:30.718ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
42302025-04-30T06:42:30.718ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42312025-04-30T06:42:30.718ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42322025-04-30T06:42:30.718ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42332025-04-30T06:42:30.718ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
42342025-04-30T06:42:30.719ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42352025-04-30T06:42:30.719ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:45353 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
42362025-04-30T06:42:30.720ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:45353 req_id = bf62d54d-880c-4726-b04f-5ea99fce0a31 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42372025-04-30T06:42:30.720ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 549 local_addr = [::1]:62607 method = GET remote_addr = [::1]:45353 req_id = bf62d54d-880c-4726-b04f-5ea99fce0a31 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42382025-04-30T06:42:30.721ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf62d54d-880c-4726-b04f-5ea99fce0a31", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
42392025-04-30T06:42:30.724ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42402025-04-30T06:42:30.724ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42412025-04-30T06:42:30.724ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
42422025-04-30T06:42:30.726ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42432025-04-30T06:42:30.726ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42442025-04-30T06:42:30.726ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42452025-04-30T06:42:30.726ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42462025-04-30T06:42:30.727ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42472025-04-30T06:42:30.727ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42482025-04-30T06:42:30.732ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42492025-04-30T06:42:30.732ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42502025-04-30T06:42:30.732ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
42512025-04-30T06:42:30.733ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42522025-04-30T06:42:30.735ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42532025-04-30T06:42:30.741ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42542025-04-30T06:42:30.742ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42552025-04-30T06:42:30.742ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42562025-04-30T06:42:30.743ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42572025-04-30T06:42:30.748ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42582025-04-30T06:42:30.748ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
42592025-04-30T06:42:30.748ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42602025-04-30T06:42:30.824ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
42612025-04-30T06:42:30.826ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
42622025-04-30T06:42:30.826ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 479.723156ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
42632025-04-30T06:42:30.826ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
42642025-04-30T06:42:30.826ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42652025-04-30T06:42:30.826ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42662025-04-30T06:42:30.826ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42672025-04-30T06:42:30.826ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42682025-04-30T06:42:30.831ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42692025-04-30T06:42:30.842ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42702025-04-30T06:42:30.846ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42712025-04-30T06:42:30.846ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42722025-04-30T06:42:30.846ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42732025-04-30T06:42:30.847ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42742025-04-30T06:42:30.855ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
42752025-04-30T06:42:30.863ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
42762025-04-30T06:42:30.863ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
42772025-04-30T06:42:30.865ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1a fd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63804 req_id = a2d15f76-01ef-4dac-b3e8-12d9a5493a19
42782025-04-30T06:42:30.865ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6909,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63804 req_id = a2d15f76-01ef-4dac-b3e8-12d9a5493a19
42792025-04-30T06:42:30.865ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
42802025-04-30T06:42:30.865ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42812025-04-30T06:42:30.867ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63804 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a2d15f76-01ef-4dac-b3e8-12d9a5493a19
42822025-04-30T06:42:30.868ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4215, tv_nsec: 667026356 } })
42832025-04-30T06:42:30.868ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
42842025-04-30T06:42:30.869ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 08 1e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64556 req_id = 884aede1-21bb-4c09-adad-67e5376f2267
42852025-04-30T06:42:30.869ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2078,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64556 req_id = 884aede1-21bb-4c09-adad-67e5376f2267
42862025-04-30T06:42:30.869ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42872025-04-30T06:42:30.870ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64556 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 884aede1-21bb-4c09-adad-67e5376f2267
42882025-04-30T06:42:30.930ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
42892025-04-30T06:42:30.930ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42902025-04-30T06:42:30.930ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42912025-04-30T06:42:30.931ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42922025-04-30T06:42:30.931ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
42932025-04-30T06:42:30.932ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42942025-04-30T06:42:30.932ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42952025-04-30T06:42:30.932ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42962025-04-30T06:42:30.932ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
42972025-04-30T06:42:30.934ZDEBGtest_delete_snapshot (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
42982025-04-30T06:42:30.934ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
42992025-04-30T06:42:30.937ZTRACtest_delete_snapshot (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
43002025-04-30T06:42:30.937ZDEBGtest_delete_snapshot (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:30.933998315Z } timeseries_name = integration_target:integration_metric
43012025-04-30T06:42:30.938ZDEBGtest_delete_snapshot (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
43022025-04-30T06:42:30.938ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
43032025-04-30T06:42:30.939ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: bb 68 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .h..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52346 req_id = 64e22399-67ae-4ae9-a912-ff5bb0c98021
43042025-04-30T06:42:30.939ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47976,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52346 req_id = 64e22399-67ae-4ae9-a912-ff5bb0c98021
43052025-04-30T06:42:30.940ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
43062025-04-30T06:42:30.940ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47976, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52346 req_id = 64e22399-67ae-4ae9-a912-ff5bb0c98021
43072025-04-30T06:42:30.940ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43082025-04-30T06:42:30.945ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43092025-04-30T06:42:30.945ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43102025-04-30T06:42:30.945ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
43112025-04-30T06:42:30.947ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43122025-04-30T06:42:30.953ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
43132025-04-30T06:42:30.953ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
43142025-04-30T06:42:30.953ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
43152025-04-30T06:42:30.956ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43162025-04-30T06:42:30.956ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
43172025-04-30T06:42:30.956ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43182025-04-30T06:42:30.959ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
43192025-04-30T06:42:30.963ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "90f93f96-511a-4525-be93-3cd93f8926b2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43202025-04-30T06:42:30.963ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
43212025-04-30T06:42:30.964ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
43222025-04-30T06:42:30.964ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
43232025-04-30T06:42:30.965ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c86c7a71-5041-4032-91ce-475560687837", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43242025-04-30T06:42:30.966ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
43252025-04-30T06:42:30.967ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
43262025-04-30T06:42:30.967ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43272025-04-30T06:42:30.969ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b410bf5-9f26-4926-84c8-b629e8512ec3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43282025-04-30T06:42:30.969ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
43292025-04-30T06:42:30.971ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2120d5a9-fd3a-4b11-b9b7-d05963adeb2d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43302025-04-30T06:42:30.971ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
43312025-04-30T06:42:30.973ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09f86a4e-f875-40cc-b45f-0366d91f3f02", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43322025-04-30T06:42:30.974ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
43332025-04-30T06:42:30.978ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bbf10e8-ff40-4755-bc56-5f650ea1609b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43342025-04-30T06:42:30.979ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
43352025-04-30T06:42:30.983ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "337822e3-d529-4a9b-b91e-5ffbc7d1cb73", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43362025-04-30T06:42:30.984ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
43372025-04-30T06:42:30.988ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f96433aa-7ce3-44d1-b2af-df22052a1ac3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43382025-04-30T06:42:30.988ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
43392025-04-30T06:42:30.990ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1993348-dab1-494c-bf76-5b4524a310d8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43402025-04-30T06:42:30.991ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
43412025-04-30T06:42:31.006ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9862ea9-630d-4215-a3ba-db7e1dc19948", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43422025-04-30T06:42:31.006ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
43432025-04-30T06:42:31.007ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
43442025-04-30T06:42:31.009ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "551be349-58d5-4b12-a7c2-479bdd405e10", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43452025-04-30T06:42:31.009ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
43462025-04-30T06:42:31.010ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a363b672-872a-4cb6-b9de-fc33316d7103", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43472025-04-30T06:42:31.010ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
43482025-04-30T06:42:31.013ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab69576f-d8ed-414a-89d2-4fe576c18337", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43492025-04-30T06:42:31.013ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
43502025-04-30T06:42:31.016ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3212f8de-e304-488c-bddc-be82cfd327f1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43512025-04-30T06:42:31.017ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
43522025-04-30T06:42:31.021ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c994a84c-0a51-494f-811a-531ba1a0f3a1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43532025-04-30T06:42:31.021ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
43542025-04-30T06:42:31.023ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d3ca1cb-9d6d-49a5-a2d1-51002fc24eb5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43552025-04-30T06:42:31.023ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
43562025-04-30T06:42:31.026ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43c14063-f543-48b0-a706-918df8656af0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43572025-04-30T06:42:31.026ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
43582025-04-30T06:42:31.028ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd1dd7e9-bea8-4457-ae61-3b9042e39231", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43592025-04-30T06:42:31.028ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
43602025-04-30T06:42:31.031ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9adcee4-737b-43c7-b748-a7da0a352b03", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43612025-04-30T06:42:31.031ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
43622025-04-30T06:42:31.033ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
43632025-04-30T06:42:31.034ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85f2bc99-b812-497e-b3b3-5d1ce691f1f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43642025-04-30T06:42:31.034ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
43652025-04-30T06:42:31.035ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee983b31-4f9c-4562-a2c9-22148e2bf3e0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43662025-04-30T06:42:31.035ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
43672025-04-30T06:42:31.036ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60c12a88-a337-44d3-9ec6-42b9ac839954", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43682025-04-30T06:42:31.036ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
43692025-04-30T06:42:31.039ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39a5d77b-210b-4f28-8b62-cb17d77ac489", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43702025-04-30T06:42:31.039ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
43712025-04-30T06:42:31.042ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27917f7d-f6da-4817-aa27-9e894c51089f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43722025-04-30T06:42:31.042ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
43732025-04-30T06:42:31.055ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d44f9d02-b139-4934-8e40-cccc2b29186c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43742025-04-30T06:42:31.055ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
43752025-04-30T06:42:31.058ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e09a7e98-b09a-471b-9f52-0a0b70547e10", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43762025-04-30T06:42:31.058ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
43772025-04-30T06:42:31.058ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43782025-04-30T06:42:31.058ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43792025-04-30T06:42:31.058ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43802025-04-30T06:42:31.058ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
43812025-04-30T06:42:31.068ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43822025-04-30T06:42:31.085ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43832025-04-30T06:42:31.101ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43842025-04-30T06:42:31.101ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43852025-04-30T06:42:31.101ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
43862025-04-30T06:42:31.102ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43872025-04-30T06:42:31.108ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43882025-04-30T06:42:31.108ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
43892025-04-30T06:42:31.110ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43902025-04-30T06:42:31.141ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0a6220e-d542-425c-ae2f-9b645d8a7563", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43912025-04-30T06:42:31.142ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
43922025-04-30T06:42:31.155ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1579a68-39a4-400c-9a1d-a067d089849b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43932025-04-30T06:42:31.155ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
43942025-04-30T06:42:31.158ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57b7d4c1-7e6f-490d-a2f8-bf9d816252f3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43952025-04-30T06:42:31.158ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
43962025-04-30T06:42:31.162ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
43972025-04-30T06:42:31.163ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b19853d-1db8-4f70-82d2-e6d8f269f883", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
43982025-04-30T06:42:31.164ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
43992025-04-30T06:42:31.165ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5f235a7-7546-40fd-b17d-f9541876f0ec", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44002025-04-30T06:42:31.165ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
44012025-04-30T06:42:31.167ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0347f114-1b14-4f4d-be4a-63a64a269382", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44022025-04-30T06:42:31.167ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
44032025-04-30T06:42:31.169ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ed0ebe3-1563-4b9c-8795-f17c8a022451", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44042025-04-30T06:42:31.170ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
44052025-04-30T06:42:31.172ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bee49cf3-5880-4eb3-be1d-586487ce04f4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44062025-04-30T06:42:31.172ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
44072025-04-30T06:42:31.179ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef20f650-7a14-4eb9-80a1-8c7631ce9af7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44082025-04-30T06:42:31.179ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
44092025-04-30T06:42:31.182ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27985c5e-495d-4aed-b375-e4cd56be3c78", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44102025-04-30T06:42:31.182ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
44112025-04-30T06:42:31.185ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aab929a3-88ec-47aa-9f4b-fc339df4943f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44122025-04-30T06:42:31.185ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
44132025-04-30T06:42:31.188ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe54755a-7255-4975-aff4-2e1b188aba9e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44142025-04-30T06:42:31.188ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
44152025-04-30T06:42:31.191ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "953b3857-d0b2-4a15-9bfa-46c9fa85ee1e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44162025-04-30T06:42:31.191ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
44172025-04-30T06:42:31.193ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba5bff64-895d-4d22-9993-c06bdab16e3a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44182025-04-30T06:42:31.194ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
44192025-04-30T06:42:31.195ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04bf5946-8198-47fd-89ef-15669119c6fc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44202025-04-30T06:42:31.195ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
44212025-04-30T06:42:31.196ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c108f8e-ec13-40e4-98bd-9e55e4784414", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44222025-04-30T06:42:31.196ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
44232025-04-30T06:42:31.197ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87c50eb5-111c-4084-9e4c-e457b9b97b05", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44242025-04-30T06:42:31.198ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
44252025-04-30T06:42:31.199ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a51eacd-78d6-496a-bc95-f0708609b23a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44262025-04-30T06:42:31.199ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
44272025-04-30T06:42:31.200ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58de932b-b46b-4ab5-a0e0-5780bc3834c5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44282025-04-30T06:42:31.200ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44292025-04-30T06:42:31.200ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44302025-04-30T06:42:31.200ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44312025-04-30T06:42:31.200ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44322025-04-30T06:42:31.201ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44332025-04-30T06:42:31.204ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44342025-04-30T06:42:31.209ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44352025-04-30T06:42:31.209ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44362025-04-30T06:42:31.209ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44372025-04-30T06:42:31.209ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44382025-04-30T06:42:31.209ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44392025-04-30T06:42:31.210ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44402025-04-30T06:42:31.213ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44412025-04-30T06:42:31.214ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44422025-04-30T06:42:31.214ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44432025-04-30T06:42:31.214ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
44442025-04-30T06:42:31.214ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44452025-04-30T06:42:31.215ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44462025-04-30T06:42:31.219ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44472025-04-30T06:42:31.219ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44482025-04-30T06:42:31.219ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44492025-04-30T06:42:31.220ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44502025-04-30T06:42:31.224ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44512025-04-30T06:42:31.228ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44522025-04-30T06:42:31.229ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44532025-04-30T06:42:31.229ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
44542025-04-30T06:42:31.229ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44552025-04-30T06:42:31.283ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
44562025-04-30T06:42:31.283ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
44572025-04-30T06:42:31.285ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:41038 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
44582025-04-30T06:42:31.286ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:41038 req_id = 092627d8-95bf-4e97-813d-d1960faccc23 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44592025-04-30T06:42:31.287ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1316 local_addr = [::1]:62607 method = GET remote_addr = [::1]:41038 req_id = 092627d8-95bf-4e97-813d-d1960faccc23 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44602025-04-30T06:42:31.288ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44612025-04-30T06:42:31.297ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
44622025-04-30T06:42:31.298ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44632025-04-30T06:42:31.298ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "092627d8-95bf-4e97-813d-d1960faccc23", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
44642025-04-30T06:42:31.301ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44652025-04-30T06:42:31.301ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44662025-04-30T06:42:31.301ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
44672025-04-30T06:42:31.305ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44682025-04-30T06:42:31.305ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44692025-04-30T06:42:31.305ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44702025-04-30T06:42:31.305ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44712025-04-30T06:42:31.305ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44722025-04-30T06:42:31.326ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44732025-04-30T06:42:31.330ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44742025-04-30T06:42:31.330ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44752025-04-30T06:42:31.330ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44762025-04-30T06:42:31.330ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44772025-04-30T06:42:31.379ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
44782025-04-30T06:42:31.381ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
44792025-04-30T06:42:31.381ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
44802025-04-30T06:42:31.381ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 555.286753ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
44812025-04-30T06:42:31.381ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
44822025-04-30T06:42:31.381ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44832025-04-30T06:42:31.382ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44842025-04-30T06:42:31.382ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44852025-04-30T06:42:31.382ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44862025-04-30T06:42:31.382ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44872025-04-30T06:42:31.384ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44882025-04-30T06:42:31.388ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44892025-04-30T06:42:31.388ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44902025-04-30T06:42:31.388ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44912025-04-30T06:42:31.388ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44922025-04-30T06:42:31.391ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
44932025-04-30T06:42:31.391ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
44942025-04-30T06:42:31.392ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cf 91 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59407 req_id = 830ec1e5-dec6-4019-a3ee-55ca91d59c05
44952025-04-30T06:42:31.392ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53137,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59407 req_id = 830ec1e5-dec6-4019-a3ee-55ca91d59c05
44962025-04-30T06:42:31.392ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
44972025-04-30T06:42:31.393ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
44982025-04-30T06:42:31.393ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59407 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 830ec1e5-dec6-4019-a3ee-55ca91d59c05
44992025-04-30T06:42:31.393ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4216, tv_nsec: 192447353 } })
45002025-04-30T06:42:31.393ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
45012025-04-30T06:42:31.393ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 35 2b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 5+..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49998 req_id = 59368292-ff9e-4e83-99bf-3ec37d3ecf49
45022025-04-30T06:42:31.394ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13611,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49998 req_id = 59368292-ff9e-4e83-99bf-3ec37d3ecf49
45032025-04-30T06:42:31.394ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45042025-04-30T06:42:31.394ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49998 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 59368292-ff9e-4e83-99bf-3ec37d3ecf49
45052025-04-30T06:42:31.446ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
45062025-04-30T06:42:31.452ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45072025-04-30T06:42:31.452ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45082025-04-30T06:42:31.452ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45092025-04-30T06:42:31.452ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
45102025-04-30T06:42:31.453ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45112025-04-30T06:42:31.454ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 25 e5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c %............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49186 req_id = 070eecea-f115-4f6d-b691-acc887a24336
45122025-04-30T06:42:31.454ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9701,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49186 req_id = 070eecea-f115-4f6d-b691-acc887a24336
45132025-04-30T06:42:31.454ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
45142025-04-30T06:42:31.455ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9701, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49186 req_id = 070eecea-f115-4f6d-b691-acc887a24336
45152025-04-30T06:42:31.456ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
45162025-04-30T06:42:31.456ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
45172025-04-30T06:42:31.456ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
45182025-04-30T06:42:31.457ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45192025-04-30T06:42:31.466ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45202025-04-30T06:42:31.466ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45212025-04-30T06:42:31.466ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
45222025-04-30T06:42:31.466ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45232025-04-30T06:42:31.468ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5db2903f-18dd-4330-b73a-ae635406c8c8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45242025-04-30T06:42:31.468ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
45252025-04-30T06:42:31.469ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed539c8c-fff5-441e-9495-ee7d43b765b5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45262025-04-30T06:42:31.470ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
45272025-04-30T06:42:31.472ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0697366-17a0-4865-865a-73799bb86383", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45282025-04-30T06:42:31.472ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
45292025-04-30T06:42:31.472ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45302025-04-30T06:42:31.472ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
45312025-04-30T06:42:31.473ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45322025-04-30T06:42:31.475ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8871acbb-eb70-4e04-8c49-7e60618c69d4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45332025-04-30T06:42:31.475ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
45342025-04-30T06:42:31.477ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66b8f91c-ddb9-41da-87e1-bc7e46a9a48c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45352025-04-30T06:42:31.477ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
45362025-04-30T06:42:31.479ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26bee7c0-b0aa-4c27-8f12-20065d8d899c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45372025-04-30T06:42:31.479ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
45382025-04-30T06:42:31.481ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b67dd2e4-a5da-491a-b622-0f1d2596e61a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45392025-04-30T06:42:31.481ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
45402025-04-30T06:42:31.484ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae7b29c5-8edd-462c-93aa-0d7273361c59", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45412025-04-30T06:42:31.484ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
45422025-04-30T06:42:31.486ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3078752e-2f7e-4a75-b32b-05175e386997", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45432025-04-30T06:42:31.486ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
45442025-04-30T06:42:31.488ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cd33f11-b1e2-403a-b098-223d6d529534", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45452025-04-30T06:42:31.488ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
45462025-04-30T06:42:31.489ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
45472025-04-30T06:42:31.490ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "889a5e84-f25d-4a46-8290-aa8cf99ad970", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45482025-04-30T06:42:31.490ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
45492025-04-30T06:42:31.490ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45502025-04-30T06:42:31.491ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = a9808dc7-8a57-487d-8acd-921cc236f2f8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45512025-04-30T06:42:31.491ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = a9808dc7-8a57-487d-8acd-921cc236f2f8 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45522025-04-30T06:42:31.491ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = a9808dc7-8a57-487d-8acd-921cc236f2f8 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45532025-04-30T06:42:31.492ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = a9808dc7-8a57-487d-8acd-921cc236f2f8 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45542025-04-30T06:42:31.495ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46c1318f-3430-4cb6-8527-17b5ce607e8b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45552025-04-30T06:42:31.496ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
45562025-04-30T06:42:31.496ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 5832 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = a9808dc7-8a57-487d-8acd-921cc236f2f8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45572025-04-30T06:42:31.498ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9808dc7-8a57-487d-8acd-921cc236f2f8", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45582025-04-30T06:42:31.498ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
45592025-04-30T06:42:31.498ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 391f385c-864e-4b0e-b874-750c590c8bb0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
45602025-04-30T06:42:31.499ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 391f385c-864e-4b0e-b874-750c590c8bb0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
45612025-04-30T06:42:31.499ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 391f385c-864e-4b0e-b874-750c590c8bb0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
45622025-04-30T06:42:31.500ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 391f385c-864e-4b0e-b874-750c590c8bb0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
45632025-04-30T06:42:31.503ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff89db4f-3457-457b-a584-d91d5705f9e2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45642025-04-30T06:42:31.504ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
45652025-04-30T06:42:31.504ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 5858 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 391f385c-864e-4b0e-b874-750c590c8bb0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
45662025-04-30T06:42:31.505ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
45672025-04-30T06:42:31.505ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "391f385c-864e-4b0e-b874-750c590c8bb0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45682025-04-30T06:42:31.506ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45692025-04-30T06:42:31.506ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
45702025-04-30T06:42:31.506ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
45712025-04-30T06:42:31.509ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b39a8d2-1bbf-47e1-8e61-563097a59fe2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45722025-04-30T06:42:31.509ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
45732025-04-30T06:42:31.512ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63007859-04d9-4760-99f1-1711ac43d2da", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45742025-04-30T06:42:31.513ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
45752025-04-30T06:42:31.514ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7b110f7-9c12-43c8-a88e-ec5a3bc926b5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45762025-04-30T06:42:31.515ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
45772025-04-30T06:42:31.518ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdaeda7d-b67f-4168-a23f-22700c59377e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45782025-04-30T06:42:31.518ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
45792025-04-30T06:42:31.520ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d96800c1-ce71-434c-8b9b-7181bcb9426d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45802025-04-30T06:42:31.520ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
45812025-04-30T06:42:31.522ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "547c3514-5035-45a2-b747-00e461d18b57", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45822025-04-30T06:42:31.522ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
45832025-04-30T06:42:31.523ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dffbc78d-c0e0-46d7-892c-ea67a00f5850", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45842025-04-30T06:42:31.523ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
45852025-04-30T06:42:31.524ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ae9e8f1-304b-41fc-aee2-745c10644d0b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45862025-04-30T06:42:31.524ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
45872025-04-30T06:42:31.525ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57c9d565-44b1-4886-9642-29fcabb7e4a0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45882025-04-30T06:42:31.525ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
45892025-04-30T06:42:31.527ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a718ae4a-dce6-4a64-bfbd-26b29e88dbeb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45902025-04-30T06:42:31.527ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
45912025-04-30T06:42:31.528ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "244ef518-1e14-4ea7-bb7f-6a0a567f385c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45922025-04-30T06:42:31.528ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
45932025-04-30T06:42:31.531ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaa841c6-a22c-4d0a-b77b-d00a7f6f0076", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45942025-04-30T06:42:31.531ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
45952025-04-30T06:42:31.533ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83cbc3fa-12c4-4b56-8888-9ce6372ea480", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45962025-04-30T06:42:31.533ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
45972025-04-30T06:42:31.542ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a374e5df-c201-4afd-849d-008ec37650fb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
45982025-04-30T06:42:31.542ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
45992025-04-30T06:42:31.548ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e9db437-1d3d-42d1-b2e1-fe20d9cedd37", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
46002025-04-30T06:42:31.549ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
46012025-04-30T06:42:31.551ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f2791d9-7c4d-4557-9ab8-678ae74a3b0b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:30 GMT"} })
46022025-04-30T06:42:31.551ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
46032025-04-30T06:42:31.552ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46042025-04-30T06:42:31.552ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46052025-04-30T06:42:31.559ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46062025-04-30T06:42:31.559ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46072025-04-30T06:42:31.559ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46082025-04-30T06:42:31.559ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
46092025-04-30T06:42:31.560ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46102025-04-30T06:42:31.561ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46112025-04-30T06:42:31.562ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46122025-04-30T06:42:31.562ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46132025-04-30T06:42:31.562ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46142025-04-30T06:42:31.565ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46152025-04-30T06:42:31.572ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46162025-04-30T06:42:31.572ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46172025-04-30T06:42:31.572ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
46182025-04-30T06:42:31.573ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46192025-04-30T06:42:31.575ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc4a6d04-a9da-448d-b61a-15446b1ff007", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46202025-04-30T06:42:31.575ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
46212025-04-30T06:42:31.576ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74391a33-26d6-4777-9ca7-f14fe02110aa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46222025-04-30T06:42:31.576ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
46232025-04-30T06:42:31.576ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46242025-04-30T06:42:31.576ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
46252025-04-30T06:42:31.577ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46262025-04-30T06:42:31.579ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6171d626-27a0-4053-8570-6e94e6cf2ec8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46272025-04-30T06:42:31.579ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
46282025-04-30T06:42:31.580ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a850911-28c2-4298-a001-0d936afd99f4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46292025-04-30T06:42:31.582ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
46302025-04-30T06:42:31.584ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f70fe210-37cd-4d4a-b703-487110fcbd41", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46312025-04-30T06:42:31.584ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
46322025-04-30T06:42:31.586ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "79c73e21-0a44-4339-b625-c8be3f1c41cd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46332025-04-30T06:42:31.586ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
46342025-04-30T06:42:31.589ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d14f009-c2d5-4023-84f5-93a21db79f9a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46352025-04-30T06:42:31.590ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
46362025-04-30T06:42:31.594ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "facfef10-f7f5-420c-9cff-275dd7db17dd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46372025-04-30T06:42:31.594ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
46382025-04-30T06:42:31.608ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8c17daa-2016-4d69-b8cb-fc2ed728f8ba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46392025-04-30T06:42:31.609ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
46402025-04-30T06:42:31.619ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a6f8543-7902-4e78-9c1a-3e90e0ae68b4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46412025-04-30T06:42:31.620ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
46422025-04-30T06:42:31.620ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
46432025-04-30T06:42:31.624ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afd8c58d-7534-486e-89b5-c8eea68365e9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46442025-04-30T06:42:31.624ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
46452025-04-30T06:42:31.628ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b9cef80-abd3-43b7-92de-08226fbe1cff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46462025-04-30T06:42:31.628ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
46472025-04-30T06:42:31.629ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec71b2ae-7c83-4e4b-9a54-e8750177ff7c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46482025-04-30T06:42:31.629ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
46492025-04-30T06:42:31.630ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bfcad59-4954-4160-866f-5d34d6f041dc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46502025-04-30T06:42:31.630ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
46512025-04-30T06:42:31.633ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b188de51-5c22-45e9-a941-7db76fa227bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46522025-04-30T06:42:31.633ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
46532025-04-30T06:42:31.636ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b6c71d4-c1b1-49bd-815d-bc856e0c5aab", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46542025-04-30T06:42:31.636ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46552025-04-30T06:42:31.636ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46562025-04-30T06:42:31.636ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46572025-04-30T06:42:31.636ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46582025-04-30T06:42:31.638ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46592025-04-30T06:42:31.645ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46602025-04-30T06:42:31.651ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46612025-04-30T06:42:31.651ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46622025-04-30T06:42:31.651ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46632025-04-30T06:42:31.651ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46642025-04-30T06:42:31.651ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46652025-04-30T06:42:31.652ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46662025-04-30T06:42:31.656ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46672025-04-30T06:42:31.662ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46682025-04-30T06:42:31.662ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46692025-04-30T06:42:31.662ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46702025-04-30T06:42:31.663ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46712025-04-30T06:42:31.733ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
46722025-04-30T06:42:31.733ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
46732025-04-30T06:42:31.734ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46742025-04-30T06:42:31.734ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46752025-04-30T06:42:31.734ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46762025-04-30T06:42:31.734ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
46772025-04-30T06:42:31.734ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46782025-04-30T06:42:31.734ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:59626 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
46792025-04-30T06:42:31.735ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:59626 req_id = b199696f-eb55-45af-be93-5616b6f3cf0f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46802025-04-30T06:42:31.736ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 614 local_addr = [::1]:62607 method = GET remote_addr = [::1]:59626 req_id = b199696f-eb55-45af-be93-5616b6f3cf0f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46812025-04-30T06:42:31.736ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b199696f-eb55-45af-be93-5616b6f3cf0f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
46822025-04-30T06:42:31.740ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46832025-04-30T06:42:31.740ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46842025-04-30T06:42:31.740ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
46852025-04-30T06:42:31.747ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46862025-04-30T06:42:31.747ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46872025-04-30T06:42:31.747ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46882025-04-30T06:42:31.747ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46892025-04-30T06:42:31.748ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46902025-04-30T06:42:31.749ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46912025-04-30T06:42:31.756ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46922025-04-30T06:42:31.756ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46932025-04-30T06:42:31.756ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
46942025-04-30T06:42:31.757ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46952025-04-30T06:42:31.759ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46962025-04-30T06:42:31.764ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
46972025-04-30T06:42:31.768ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46982025-04-30T06:42:31.768ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46992025-04-30T06:42:31.768ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47002025-04-30T06:42:31.773ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47012025-04-30T06:42:31.773ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
47022025-04-30T06:42:31.773ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47032025-04-30T06:42:31.804ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
47042025-04-30T06:42:31.805ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
47052025-04-30T06:42:31.805ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 423.530787ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
47062025-04-30T06:42:31.805ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
47072025-04-30T06:42:31.805ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
47082025-04-30T06:42:31.805ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47092025-04-30T06:42:31.806ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47102025-04-30T06:42:31.806ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47112025-04-30T06:42:31.808ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47122025-04-30T06:42:31.811ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47132025-04-30T06:42:31.820ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47142025-04-30T06:42:31.820ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47152025-04-30T06:42:31.820ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47162025-04-30T06:42:31.821ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47172025-04-30T06:42:31.822ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
47182025-04-30T06:42:31.829ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
47192025-04-30T06:42:31.830ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47202025-04-30T06:42:31.830ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 59 ff 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Y............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65145 req_id = 1f02e4c5-d52e-4b2c-83b9-dbea5f7d56d6
47212025-04-30T06:42:31.831ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23039,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65145 req_id = 1f02e4c5-d52e-4b2c-83b9-dbea5f7d56d6
47222025-04-30T06:42:31.831ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
47232025-04-30T06:42:31.832ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47242025-04-30T06:42:31.832ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65145 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1f02e4c5-d52e-4b2c-83b9-dbea5f7d56d6
47252025-04-30T06:42:31.839ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4216, tv_nsec: 631659504 } })
47262025-04-30T06:42:31.840ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
47272025-04-30T06:42:31.840ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9a 4d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .M..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56806 req_id = 3075add1-5429-49e3-aa50-88c16e4ff69a
47282025-04-30T06:42:31.842ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39501,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56806 req_id = 3075add1-5429-49e3-aa50-88c16e4ff69a
47292025-04-30T06:42:31.842ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47302025-04-30T06:42:31.842ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56806 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3075add1-5429-49e3-aa50-88c16e4ff69a
47312025-04-30T06:42:31.891ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47322025-04-30T06:42:31.891ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47332025-04-30T06:42:31.891ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47342025-04-30T06:42:31.892ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47352025-04-30T06:42:31.892ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
47362025-04-30T06:42:31.894ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47372025-04-30T06:42:31.895ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d8 56 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .V..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64638 req_id = 592237e5-2d34-43fb-9e8f-1c1399ed800a
47382025-04-30T06:42:31.895ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55382,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64638 req_id = 592237e5-2d34-43fb-9e8f-1c1399ed800a
47392025-04-30T06:42:31.895ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
47402025-04-30T06:42:31.895ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55382, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64638 req_id = 592237e5-2d34-43fb-9e8f-1c1399ed800a
47412025-04-30T06:42:31.896ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
47422025-04-30T06:42:31.896ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
47432025-04-30T06:42:31.896ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
47442025-04-30T06:42:31.897ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47452025-04-30T06:42:31.903ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47462025-04-30T06:42:31.903ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47472025-04-30T06:42:31.903ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
47482025-04-30T06:42:31.904ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47492025-04-30T06:42:31.905ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47502025-04-30T06:42:31.905ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47512025-04-30T06:42:31.905ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
47522025-04-30T06:42:31.906ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
47532025-04-30T06:42:31.907ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dd03d06-ee8f-4478-96bc-47ff1af54d00", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47542025-04-30T06:42:31.907ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
47552025-04-30T06:42:31.908ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47562025-04-30T06:42:31.908ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
47572025-04-30T06:42:31.908ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47582025-04-30T06:42:31.911ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e236a7c-bed9-46f9-af4f-030411d08431", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47592025-04-30T06:42:31.911ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
47602025-04-30T06:42:31.912ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
47612025-04-30T06:42:31.913ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
47622025-04-30T06:42:31.922ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
47632025-04-30T06:42:31.922ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47642025-04-30T06:42:31.923ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92f0f3b4-cd77-4147-b3c4-e25017e5b0ee", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47652025-04-30T06:42:31.923ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
47662025-04-30T06:42:31.926ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "980be005-1859-4f4e-89ae-dad1407976ae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47672025-04-30T06:42:31.927ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
47682025-04-30T06:42:31.931ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e13240f0-5a47-434e-88f5-014bdaf9b9f9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47692025-04-30T06:42:31.931ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
47702025-04-30T06:42:31.935ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2ca3eb5-2a0a-4fbe-a743-a9d5d0527a87", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47712025-04-30T06:42:31.935ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
47722025-04-30T06:42:31.942ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f6f6026-774c-4875-9d8b-a0d0e33fd9e6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47732025-04-30T06:42:31.944ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
47742025-04-30T06:42:31.951ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9362e1ca-836d-4b5a-9efc-b69788e8c482", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47752025-04-30T06:42:31.952ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
47762025-04-30T06:42:31.956ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
47772025-04-30T06:42:31.958ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0d5efe1-9ad2-4700-9d81-f740491ba7e1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47782025-04-30T06:42:31.958ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
47792025-04-30T06:42:31.960ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9060238-5144-45cb-bc5a-1ca8910b72bf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47802025-04-30T06:42:31.960ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
47812025-04-30T06:42:31.962ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c42b8ae-2797-4c4a-9488-8e445d9d3f77", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47822025-04-30T06:42:31.962ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
47832025-04-30T06:42:31.963ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd763ada-87d2-4ffc-802a-00761a25a0fc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47842025-04-30T06:42:31.963ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
47852025-04-30T06:42:31.965ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32339f24-1bc1-48aa-ac6d-d4574e776e08", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47862025-04-30T06:42:31.965ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
47872025-04-30T06:42:31.969ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bc61f59-ec30-4f8e-80ca-2b68e11e85ef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47882025-04-30T06:42:31.969ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
47892025-04-30T06:42:31.972ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfbfee94-79d6-42d3-a9d6-5686cc642b54", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47902025-04-30T06:42:31.973ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
47912025-04-30T06:42:31.975ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00542f79-e188-4380-87bc-186f1145d491", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47922025-04-30T06:42:31.975ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
47932025-04-30T06:42:31.978ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cc1e491-1a87-4757-8c38-c16a6669ce41", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47942025-04-30T06:42:31.978ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
47952025-04-30T06:42:31.980ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c42e46cb-d4b9-4e2f-97c2-bc3c1b79e8d5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47962025-04-30T06:42:31.980ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
47972025-04-30T06:42:31.982ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1935c754-b280-4690-9ad4-b9744f8e3621", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
47982025-04-30T06:42:31.982ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
47992025-04-30T06:42:31.983ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6652dc74-5337-45d0-bcd9-72514a1e12c4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48002025-04-30T06:42:31.983ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
48012025-04-30T06:42:31.985ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67e0094b-2131-45fb-8904-f15132683f2f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48022025-04-30T06:42:31.985ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
48032025-04-30T06:42:31.986ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98117db1-13cd-49b4-b33b-23e1bbbf2d6b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48042025-04-30T06:42:31.986ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
48052025-04-30T06:42:31.987ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d91332a4-a8d8-4251-acbd-266bac4352ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48062025-04-30T06:42:31.987ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
48072025-04-30T06:42:31.989ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d189052-2b0f-4ebf-8cd6-d1bb01fbac20", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48082025-04-30T06:42:31.989ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
48092025-04-30T06:42:31.992ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d19d70b-3e28-4d4b-8089-f1776e5c9476", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48102025-04-30T06:42:31.992ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
48112025-04-30T06:42:31.994ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d796ae5-7dbe-447a-8de4-38da983d54c4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48122025-04-30T06:42:31.994ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
48132025-04-30T06:42:31.997ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef64022a-c5e3-4253-833c-deef4b59942b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48142025-04-30T06:42:31.997ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
48152025-04-30T06:42:31.999ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7e60474-7ae6-436e-bc0c-efcd0515f839", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48162025-04-30T06:42:31.999ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
48172025-04-30T06:42:32.002ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23ae3923-0702-419e-a23d-912ab1c7ecc8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48182025-04-30T06:42:32.002ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
48192025-04-30T06:42:32.005ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdb6f7bc-002c-42b6-80e6-371001359c8f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48202025-04-30T06:42:32.005ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
48212025-04-30T06:42:32.006ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
48222025-04-30T06:42:32.007ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cda1e07f-5ec5-43ea-8d9f-ab4312a9ff04", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48232025-04-30T06:42:32.007ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
48242025-04-30T06:42:32.008ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48252025-04-30T06:42:32.008ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48262025-04-30T06:42:32.009ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48272025-04-30T06:42:32.009ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
48282025-04-30T06:42:32.010ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48292025-04-30T06:42:32.011ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee3f461d-cec8-4538-bdef-36c6676ef9fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48302025-04-30T06:42:32.011ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
48312025-04-30T06:42:32.012ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "503b70f6-0dcc-4bd1-915f-46e128f6af87", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48322025-04-30T06:42:32.012ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
48332025-04-30T06:42:32.012ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48342025-04-30T06:42:32.018ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48352025-04-30T06:42:32.018ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48362025-04-30T06:42:32.019ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
48372025-04-30T06:42:32.019ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48382025-04-30T06:42:32.021ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "719d1917-9dc8-409a-b47d-b4964946c8d2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48392025-04-30T06:42:32.021ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
48402025-04-30T06:42:32.023ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48412025-04-30T06:42:32.023ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
48422025-04-30T06:42:32.024ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48432025-04-30T06:42:32.024ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf1ba0e7-4163-41d1-942d-088481a0c313", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48442025-04-30T06:42:32.024ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
48452025-04-30T06:42:32.028ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a916507-9014-4387-884b-d08d997da944", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48462025-04-30T06:42:32.029ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
48472025-04-30T06:42:32.031ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f984d09-30a3-4d10-87a2-73d32778d0d0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48482025-04-30T06:42:32.031ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
48492025-04-30T06:42:32.035ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a35dc98-7af8-482a-82d6-fe690636020c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48502025-04-30T06:42:32.036ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
48512025-04-30T06:42:32.039ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81edb421-04ec-4562-b3ed-352c500094a9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48522025-04-30T06:42:32.039ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
48532025-04-30T06:42:32.042ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac119d65-531d-4c7c-988f-2a28a6e2a38b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48542025-04-30T06:42:32.042ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
48552025-04-30T06:42:32.045ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33a19297-d847-48a3-a7af-156d06acc8bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48562025-04-30T06:42:32.045ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
48572025-04-30T06:42:32.046ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f670e21-f680-4f45-a959-1e9597537019", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48582025-04-30T06:42:32.046ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
48592025-04-30T06:42:32.048ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20d1d60b-b7b5-4b20-968d-94e138b020d2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48602025-04-30T06:42:32.048ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
48612025-04-30T06:42:32.050ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa4f68b1-4fa3-4b2a-9838-fc1411af263a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48622025-04-30T06:42:32.050ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
48632025-04-30T06:42:32.054ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f26d6120-b8ab-4539-b4af-ee3e2e2119e4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48642025-04-30T06:42:32.054ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48652025-04-30T06:42:32.054ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48662025-04-30T06:42:32.054ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48672025-04-30T06:42:32.055ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48682025-04-30T06:42:32.055ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48692025-04-30T06:42:32.057ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
48702025-04-30T06:42:32.059ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48712025-04-30T06:42:32.067ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48722025-04-30T06:42:32.067ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48732025-04-30T06:42:32.067ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48742025-04-30T06:42:32.067ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48752025-04-30T06:42:32.067ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48762025-04-30T06:42:32.068ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48772025-04-30T06:42:32.082ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48782025-04-30T06:42:32.086ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48792025-04-30T06:42:32.087ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48802025-04-30T06:42:32.087ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48812025-04-30T06:42:32.088ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48822025-04-30T06:42:32.220ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
48832025-04-30T06:42:32.220ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
48842025-04-30T06:42:32.220ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48852025-04-30T06:42:32.220ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48862025-04-30T06:42:32.220ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48872025-04-30T06:42:32.220ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
48882025-04-30T06:42:32.221ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48892025-04-30T06:42:32.222ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:56739 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
48902025-04-30T06:42:32.223ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:56739 req_id = 3d40dc65-b886-4fa3-bf74-ac9821040f01 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48912025-04-30T06:42:32.227ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 3911 local_addr = [::1]:62607 method = GET remote_addr = [::1]:56739 req_id = 3d40dc65-b886-4fa3-bf74-ac9821040f01 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48922025-04-30T06:42:32.229ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d40dc65-b886-4fa3-bf74-ac9821040f01", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
48932025-04-30T06:42:32.234ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48942025-04-30T06:42:32.234ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48952025-04-30T06:42:32.234ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
48962025-04-30T06:42:32.236ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
48972025-04-30T06:42:32.236ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48982025-04-30T06:42:32.239ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48992025-04-30T06:42:32.239ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49002025-04-30T06:42:32.240ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49012025-04-30T06:42:32.240ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49022025-04-30T06:42:32.245ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49032025-04-30T06:42:32.245ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49042025-04-30T06:42:32.246ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
49052025-04-30T06:42:32.246ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49062025-04-30T06:42:32.250ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49072025-04-30T06:42:32.254ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49082025-04-30T06:42:32.254ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49092025-04-30T06:42:32.254ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49102025-04-30T06:42:32.255ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49112025-04-30T06:42:32.259ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49122025-04-30T06:42:32.259ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
49132025-04-30T06:42:32.259ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49142025-04-30T06:42:32.378ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
49152025-04-30T06:42:32.378ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
49162025-04-30T06:42:32.378ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 572.88723ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49172025-04-30T06:42:32.378ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
49182025-04-30T06:42:32.378ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49192025-04-30T06:42:32.378ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49202025-04-30T06:42:32.379ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49212025-04-30T06:42:32.379ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49222025-04-30T06:42:32.379ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49232025-04-30T06:42:32.383ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49242025-04-30T06:42:32.396ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49252025-04-30T06:42:32.396ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49262025-04-30T06:42:32.396ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49272025-04-30T06:42:32.397ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49282025-04-30T06:42:32.419ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
49292025-04-30T06:42:32.422ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
49302025-04-30T06:42:32.423ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49312025-04-30T06:42:32.423ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0d 48 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .H..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49519 req_id = 2a0dd9eb-d0e0-4779-9ddb-856973635126
49322025-04-30T06:42:32.423ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3400,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49519 req_id = 2a0dd9eb-d0e0-4779-9ddb-856973635126
49332025-04-30T06:42:32.423ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
49342025-04-30T06:42:32.424ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49352025-04-30T06:42:32.424ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49519 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2a0dd9eb-d0e0-4779-9ddb-856973635126
49362025-04-30T06:42:32.424ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4217, tv_nsec: 223478766 } })
49372025-04-30T06:42:32.424ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
49382025-04-30T06:42:32.424ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0f a2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55530 req_id = 2b1a7e3e-15ad-417c-98d6-5a4532ef6582
49392025-04-30T06:42:32.425ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4002,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55530 req_id = 2b1a7e3e-15ad-417c-98d6-5a4532ef6582
49402025-04-30T06:42:32.425ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49412025-04-30T06:42:32.425ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55530 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2b1a7e3e-15ad-417c-98d6-5a4532ef6582
49422025-04-30T06:42:32.495ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49432025-04-30T06:42:32.495ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49442025-04-30T06:42:32.495ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49452025-04-30T06:42:32.495ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49462025-04-30T06:42:32.496ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
49472025-04-30T06:42:32.496ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49482025-04-30T06:42:32.497ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ab 09 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53842 req_id = bf3b9db0-02d4-4fc7-b5e0-48bbbfa10842
49492025-04-30T06:42:32.497ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43785,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53842 req_id = bf3b9db0-02d4-4fc7-b5e0-48bbbfa10842
49502025-04-30T06:42:32.497ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
49512025-04-30T06:42:32.497ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43785, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53842 req_id = bf3b9db0-02d4-4fc7-b5e0-48bbbfa10842
49522025-04-30T06:42:32.497ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
49532025-04-30T06:42:32.498ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
49542025-04-30T06:42:32.498ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
49552025-04-30T06:42:32.499ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49562025-04-30T06:42:32.505ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49572025-04-30T06:42:32.505ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49582025-04-30T06:42:32.505ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
49592025-04-30T06:42:32.506ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49602025-04-30T06:42:32.513ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49612025-04-30T06:42:32.513ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
49622025-04-30T06:42:32.514ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49632025-04-30T06:42:32.515ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8abb49c4-aa4c-418d-b980-243997d2cee7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
49642025-04-30T06:42:32.515ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
49652025-04-30T06:42:32.519ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f284e424-1597-4aa1-a893-bef12f6d7036", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
49662025-04-30T06:42:32.520ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
49672025-04-30T06:42:32.555ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49682025-04-30T06:42:32.555ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49692025-04-30T06:42:32.565ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49702025-04-30T06:42:32.566ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49712025-04-30T06:42:32.566ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49722025-04-30T06:42:32.567ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49732025-04-30T06:42:32.568ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8926dca7-589d-4acb-8b31-407b00952fcd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:31 GMT"} })
49742025-04-30T06:42:32.568ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
49752025-04-30T06:42:32.575ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77f6839d-a5c5-4981-989b-057d7068c0ab", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49762025-04-30T06:42:32.575ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
49772025-04-30T06:42:32.590ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb45ca03-0895-483e-9cdc-edd3d4d97995", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49782025-04-30T06:42:32.590ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
49792025-04-30T06:42:32.592ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
49802025-04-30T06:42:32.603ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5fc543a-6f8d-4883-bae2-b4effd3846a5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49812025-04-30T06:42:32.603ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
49822025-04-30T06:42:32.610ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0ab01ca-0a37-400f-8e48-ea8eef707df9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49832025-04-30T06:42:32.611ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
49842025-04-30T06:42:32.615ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a37c189-4290-40f2-be36-58fc40d43acf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49852025-04-30T06:42:32.615ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
49862025-04-30T06:42:32.616ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9e2f5b1-b217-4c89-9818-628d202816b3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49872025-04-30T06:42:32.616ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
49882025-04-30T06:42:32.620ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62223973-4c1a-4d65-b645-4b55fbc79437", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49892025-04-30T06:42:32.621ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
49902025-04-30T06:42:32.622ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "deb66c13-2dfc-4195-9034-25dd18c5e031", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49912025-04-30T06:42:32.622ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
49922025-04-30T06:42:32.623ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03b67eeb-3ad3-4538-b3a1-2e9b84b66fb2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49932025-04-30T06:42:32.623ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
49942025-04-30T06:42:32.626ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c2cc1c2-a5df-4d0e-bb55-a4e09773c9ef", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49952025-04-30T06:42:32.628ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
49962025-04-30T06:42:32.631ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b60548b7-7e8f-4d31-91dd-52ce4d8be711", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49972025-04-30T06:42:32.631ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
49982025-04-30T06:42:32.634ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83adac6b-7051-4a51-aef7-8b64d9884231", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
49992025-04-30T06:42:32.634ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
50002025-04-30T06:42:32.643ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50012025-04-30T06:42:32.643ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50022025-04-30T06:42:32.644ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50032025-04-30T06:42:32.644ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
50042025-04-30T06:42:32.646ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50052025-04-30T06:42:32.647ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "020dcb05-68ea-4895-9faa-11b1bb8ae6b3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50062025-04-30T06:42:32.647ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
50072025-04-30T06:42:32.651ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50082025-04-30T06:42:32.659ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50092025-04-30T06:42:32.659ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50102025-04-30T06:42:32.660ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
50112025-04-30T06:42:32.661ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50122025-04-30T06:42:32.661ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a4950c2-715c-47df-9b52-8d7aadf0f1ba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50132025-04-30T06:42:32.661ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
50142025-04-30T06:42:32.664ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50152025-04-30T06:42:32.664ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
50162025-04-30T06:42:32.665ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50172025-04-30T06:42:32.671ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ce788e4-c50d-4d8f-b798-ea21c21e50ce", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50182025-04-30T06:42:32.672ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
50192025-04-30T06:42:32.679ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43c6326c-5756-48a3-9f28-0e21e263a2c4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50202025-04-30T06:42:32.680ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
50212025-04-30T06:42:32.712ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "544bed37-b095-4599-9b78-64d1d5239888", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50222025-04-30T06:42:32.727ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
50232025-04-30T06:42:32.734ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8455794a-c3a4-4174-b0be-87593239191c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50242025-04-30T06:42:32.734ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
50252025-04-30T06:42:32.736ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "764f07dd-f78c-45a7-8921-0c0beef4e7e8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50262025-04-30T06:42:32.736ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
50272025-04-30T06:42:32.739ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2519ed79-ae69-4b4a-bbc7-f1dcb780e0c9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50282025-04-30T06:42:32.739ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
50292025-04-30T06:42:32.741ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78d33f38-4fc8-41b2-a5f3-3708561b00d5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50302025-04-30T06:42:32.742ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
50312025-04-30T06:42:32.745ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fc13a02-be73-44cb-a8db-8cc86324f27c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50322025-04-30T06:42:32.745ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
50332025-04-30T06:42:32.747ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7ddebf4-d165-46fb-af9f-fe3a20b2c442", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50342025-04-30T06:42:32.747ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
50352025-04-30T06:42:32.752ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
50362025-04-30T06:42:32.756ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23e19eec-f05a-44cc-acd9-f9e6a74dcb73", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50372025-04-30T06:42:32.756ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
50382025-04-30T06:42:32.760ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2324f489-d435-4812-bd26-8b390fceeffa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50392025-04-30T06:42:32.760ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
50402025-04-30T06:42:32.763ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ff3d381-095a-4840-9466-6574cd03316f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50412025-04-30T06:42:32.763ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
50422025-04-30T06:42:32.766ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b72229ed-b72b-4d11-9f4e-02c40ebc5d2e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50432025-04-30T06:42:32.766ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
50442025-04-30T06:42:32.767ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38418dea-ffaa-4e93-a0c4-bbf9f69d2700", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50452025-04-30T06:42:32.778ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
50462025-04-30T06:42:32.803ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18b616dd-999b-453b-81a1-2f9fe5ae2668", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50472025-04-30T06:42:32.803ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
50482025-04-30T06:42:32.805ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50492025-04-30T06:42:32.806ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50502025-04-30T06:42:32.811ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50512025-04-30T06:42:32.811ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
50522025-04-30T06:42:32.812ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50532025-04-30T06:42:32.815ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f135400-283d-4626-9d86-fbb2ef31cbef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50542025-04-30T06:42:32.815ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
50552025-04-30T06:42:32.816ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f200bb8d-73d4-48b7-b038-397c04b539b9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50562025-04-30T06:42:32.816ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
50572025-04-30T06:42:32.817ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50582025-04-30T06:42:32.823ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50592025-04-30T06:42:32.824ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50602025-04-30T06:42:32.824ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
50612025-04-30T06:42:32.824ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50622025-04-30T06:42:32.826ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a760de0-9ba6-4049-b10d-ed73f1ce9fd4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50632025-04-30T06:42:32.827ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
50642025-04-30T06:42:32.828ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50652025-04-30T06:42:32.828ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
50662025-04-30T06:42:32.828ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50672025-04-30T06:42:32.835ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c4e12b7-12f5-4bc6-b48b-f3162aeb9680", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50682025-04-30T06:42:32.835ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
50692025-04-30T06:42:32.846ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c38b64b-ffd8-41f1-b4c9-acb410849605", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50702025-04-30T06:42:32.847ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
50712025-04-30T06:42:32.851ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fdc6f5e-7b57-47b7-999d-bd301969b3b4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50722025-04-30T06:42:32.852ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
50732025-04-30T06:42:32.856ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfcc50e4-c8bd-441a-bd7e-91fd02fe2a88", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50742025-04-30T06:42:32.856ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
50752025-04-30T06:42:32.863ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f000557-25fb-4426-8fa2-b6ab9033a3d1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50762025-04-30T06:42:32.864ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
50772025-04-30T06:42:32.864ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
50782025-04-30T06:42:32.867ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "222022de-8791-4d03-af6e-5d5c369dd002", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50792025-04-30T06:42:32.867ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
50802025-04-30T06:42:32.869ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f992b66-e4ce-4583-94bf-b91b94e83f90", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50812025-04-30T06:42:32.869ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
50822025-04-30T06:42:32.874ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2921233-2ff8-4f40-88cc-24443b38e426", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50832025-04-30T06:42:32.874ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
50842025-04-30T06:42:32.876ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d403e5f1-008c-423d-874d-96c631e04207", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50852025-04-30T06:42:32.876ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
50862025-04-30T06:42:32.877ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05dca30e-525c-41c9-b118-37adbc67ebae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
50872025-04-30T06:42:32.877ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50882025-04-30T06:42:32.878ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50892025-04-30T06:42:32.878ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50902025-04-30T06:42:32.878ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50912025-04-30T06:42:32.878ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50922025-04-30T06:42:32.882ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50932025-04-30T06:42:32.909ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50942025-04-30T06:42:32.909ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50952025-04-30T06:42:32.909ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50962025-04-30T06:42:32.909ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50972025-04-30T06:42:32.909ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50982025-04-30T06:42:32.910ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50992025-04-30T06:42:32.910ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51002025-04-30T06:42:32.910ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51012025-04-30T06:42:32.911ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
51022025-04-30T06:42:32.911ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
51032025-04-30T06:42:32.923ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51042025-04-30T06:42:32.923ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51052025-04-30T06:42:32.923ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51062025-04-30T06:42:32.923ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
51072025-04-30T06:42:32.924ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51082025-04-30T06:42:32.925ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51092025-04-30T06:42:32.931ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51102025-04-30T06:42:32.931ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51112025-04-30T06:42:32.931ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51122025-04-30T06:42:32.932ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51132025-04-30T06:42:32.945ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51142025-04-30T06:42:32.952ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51152025-04-30T06:42:32.952ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51162025-04-30T06:42:32.952ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
51172025-04-30T06:42:32.952ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51182025-04-30T06:42:32.953ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
51192025-04-30T06:42:32.953ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
51202025-04-30T06:42:33.017ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
51212025-04-30T06:42:33.017ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
51222025-04-30T06:42:33.024ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:45161 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
51232025-04-30T06:42:33.024ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
51242025-04-30T06:42:33.025ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:45161 req_id = d9477448-f7f4-4dce-9b0f-c903d362ce98 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51252025-04-30T06:42:33.025ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 454 local_addr = [::1]:62607 method = GET remote_addr = [::1]:45161 req_id = d9477448-f7f4-4dce-9b0f-c903d362ce98 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51262025-04-30T06:42:33.026ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51272025-04-30T06:42:33.026ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
51282025-04-30T06:42:33.026ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51292025-04-30T06:42:33.026ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9477448-f7f4-4dce-9b0f-c903d362ce98", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
51302025-04-30T06:42:33.028ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51312025-04-30T06:42:33.028ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51322025-04-30T06:42:33.029ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
51332025-04-30T06:42:33.035ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51342025-04-30T06:42:33.035ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51352025-04-30T06:42:33.035ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51362025-04-30T06:42:33.035ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51372025-04-30T06:42:33.036ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51382025-04-30T06:42:33.039ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
51392025-04-30T06:42:33.040ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51402025-04-30T06:42:33.049ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51412025-04-30T06:42:33.055ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51422025-04-30T06:42:33.055ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51432025-04-30T06:42:33.055ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51442025-04-30T06:42:33.055ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51452025-04-30T06:42:33.114ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
51462025-04-30T06:42:33.122ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
51472025-04-30T06:42:33.122ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
51482025-04-30T06:42:33.123ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 744.69711ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
51492025-04-30T06:42:33.123ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
51502025-04-30T06:42:33.123ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51512025-04-30T06:42:33.123ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51522025-04-30T06:42:33.123ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51532025-04-30T06:42:33.123ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51542025-04-30T06:42:33.124ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51552025-04-30T06:42:33.137ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51562025-04-30T06:42:33.141ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51572025-04-30T06:42:33.141ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51582025-04-30T06:42:33.142ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51592025-04-30T06:42:33.142ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51602025-04-30T06:42:33.150ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
51612025-04-30T06:42:33.150ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
51622025-04-30T06:42:33.152ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c3 cb 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55482 req_id = dd629e62-06c6-4e94-9de3-ee77410fe3b5
51632025-04-30T06:42:33.152ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50123,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55482 req_id = dd629e62-06c6-4e94-9de3-ee77410fe3b5
51642025-04-30T06:42:33.153ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
51652025-04-30T06:42:33.163ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51662025-04-30T06:42:33.163ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55482 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = dd629e62-06c6-4e94-9de3-ee77410fe3b5
51672025-04-30T06:42:33.163ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4217, tv_nsec: 956202330 } })
51682025-04-30T06:42:33.163ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
51692025-04-30T06:42:33.163ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ba ff 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59308 req_id = 867ec2d4-3155-42ce-aea1-40a563254d60
51702025-04-30T06:42:33.163ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47871,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59308 req_id = 867ec2d4-3155-42ce-aea1-40a563254d60
51712025-04-30T06:42:33.163ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51722025-04-30T06:42:33.163ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59308 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 867ec2d4-3155-42ce-aea1-40a563254d60
51732025-04-30T06:42:33.258ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
51742025-04-30T06:42:33.259ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51752025-04-30T06:42:33.259ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51762025-04-30T06:42:33.265ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51772025-04-30T06:42:33.265ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
51782025-04-30T06:42:33.267ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51792025-04-30T06:42:33.272ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 02 bf 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54896 req_id = 3e672e5c-addf-4640-b283-a82d56864b0f
51802025-04-30T06:42:33.272ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 703,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54896 req_id = 3e672e5c-addf-4640-b283-a82d56864b0f
51812025-04-30T06:42:33.273ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
51822025-04-30T06:42:33.273ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 703, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54896 req_id = 3e672e5c-addf-4640-b283-a82d56864b0f
51832025-04-30T06:42:33.273ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
51842025-04-30T06:42:33.274ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
51852025-04-30T06:42:33.280ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
51862025-04-30T06:42:33.304ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51872025-04-30T06:42:33.328ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51882025-04-30T06:42:33.329ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51892025-04-30T06:42:33.329ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
51902025-04-30T06:42:33.330ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51912025-04-30T06:42:33.334ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2716f446-d8ee-40be-b727-f9966cf41d92", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
51922025-04-30T06:42:33.334ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
51932025-04-30T06:42:33.336ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51942025-04-30T06:42:33.336ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
51952025-04-30T06:42:33.337ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51962025-04-30T06:42:33.338ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5872906f-812e-4cd7-a5c0-4269f09d3320", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
51972025-04-30T06:42:33.339ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
51982025-04-30T06:42:33.341ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1b76728-bf4e-42a4-9bad-748b2b7ce643", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
51992025-04-30T06:42:33.341ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
52002025-04-30T06:42:33.350ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76d164c2-c0f6-411a-9418-99e84d6a7237", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52012025-04-30T06:42:33.359ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
52022025-04-30T06:42:33.369ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5e3996d-eeb3-4df3-ad22-f3d3bb41f5c8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52032025-04-30T06:42:33.370ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
52042025-04-30T06:42:33.372ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d50bdcc7-445b-4236-9761-bc1976fe963a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52052025-04-30T06:42:33.372ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
52062025-04-30T06:42:33.374ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64884523-21d4-47c3-8dc9-0232df5e013e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52072025-04-30T06:42:33.374ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
52082025-04-30T06:42:33.376ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42fec63e-8f9d-441f-9af5-765e9241c06b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52092025-04-30T06:42:33.376ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
52102025-04-30T06:42:33.377ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f532b5a8-cf79-4ca2-86e5-53cd5bc0f989", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52112025-04-30T06:42:33.377ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
52122025-04-30T06:42:33.378ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62bf28cb-c652-4353-9fd1-10348e79212e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52132025-04-30T06:42:33.378ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
52142025-04-30T06:42:33.379ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "123f9149-1d7e-4b92-acd7-fc12f49c27ac", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52152025-04-30T06:42:33.379ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
52162025-04-30T06:42:33.380ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bdb46ed-b7ca-48e2-8668-c8377607906b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52172025-04-30T06:42:33.380ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
52182025-04-30T06:42:33.384ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd8c4fcd-9b02-42c6-804e-d2f6abcefb70", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52192025-04-30T06:42:33.385ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
52202025-04-30T06:42:33.391ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd1cb73e-0a6a-446f-a52f-39fb673b45ad", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52212025-04-30T06:42:33.392ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
52222025-04-30T06:42:33.396ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb862fc-66c4-4774-bfe8-97d36d68f8db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52232025-04-30T06:42:33.397ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
52242025-04-30T06:42:33.411ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
52252025-04-30T06:42:33.414ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2cc690f-b0e5-427a-b77c-9604b13c974d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52262025-04-30T06:42:33.415ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
52272025-04-30T06:42:33.419ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e05c197d-135f-4bbe-a614-0a428922f785", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52282025-04-30T06:42:33.419ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
52292025-04-30T06:42:33.425ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4f763c9-2067-46b0-a59d-c354457e2fd8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52302025-04-30T06:42:33.425ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
52312025-04-30T06:42:33.427ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc9ac055-52a3-4342-b7a3-eafd72802db1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52322025-04-30T06:42:33.427ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
52332025-04-30T06:42:33.429ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df34d933-bebf-4575-9d2f-43faf2d30a6a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52342025-04-30T06:42:33.429ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
52352025-04-30T06:42:33.431ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50167b84-a4eb-4e4d-8ddc-9c345032c40e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52362025-04-30T06:42:33.431ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
52372025-04-30T06:42:33.433ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b97b7ce-fbb7-44ca-85f1-e295b4a95bf9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52382025-04-30T06:42:33.433ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
52392025-04-30T06:42:33.436ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c457e00e-017b-41d4-95aa-1700d088a531", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52402025-04-30T06:42:33.436ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
52412025-04-30T06:42:33.437ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a70b31a-71ac-4bf5-a94c-4bc592f19a53", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52422025-04-30T06:42:33.438ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
52432025-04-30T06:42:33.453ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c35a29d9-f673-4387-86ba-d6eeeea8bf48", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52442025-04-30T06:42:33.453ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
52452025-04-30T06:42:33.457ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20ab079c-abbd-436e-844b-c0035e2c82c2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52462025-04-30T06:42:33.457ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
52472025-04-30T06:42:33.461ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a98d7fa8-f2f7-433b-8ca3-140271d5ac3c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52482025-04-30T06:42:33.462ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
52492025-04-30T06:42:33.464ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52502025-04-30T06:42:33.464ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52512025-04-30T06:42:33.464ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52522025-04-30T06:42:33.464ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
52532025-04-30T06:42:33.465ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52542025-04-30T06:42:33.468ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52552025-04-30T06:42:33.477ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52562025-04-30T06:42:33.478ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52572025-04-30T06:42:33.478ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
52582025-04-30T06:42:33.478ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52592025-04-30T06:42:33.479ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "802afb71-2221-465c-b1ae-ff4cdbfb8835", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52602025-04-30T06:42:33.479ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
52612025-04-30T06:42:33.481ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52622025-04-30T06:42:33.481ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
52632025-04-30T06:42:33.482ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52642025-04-30T06:42:33.489ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
52652025-04-30T06:42:33.490ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "24806f60-2830-471a-9732-48515deab9d7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52662025-04-30T06:42:33.491ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
52672025-04-30T06:42:33.491ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52682025-04-30T06:42:33.493ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 029ce1ad-1c10-47c4-baa8-07a23b9692d3 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52692025-04-30T06:42:33.493ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 029ce1ad-1c10-47c4-baa8-07a23b9692d3 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52702025-04-30T06:42:33.494ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 029ce1ad-1c10-47c4-baa8-07a23b9692d3 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52712025-04-30T06:42:33.494ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 029ce1ad-1c10-47c4-baa8-07a23b9692d3 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52722025-04-30T06:42:33.499ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 5854 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 029ce1ad-1c10-47c4-baa8-07a23b9692d3 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52732025-04-30T06:42:33.502ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "029ce1ad-1c10-47c4-baa8-07a23b9692d3", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52742025-04-30T06:42:33.502ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
52752025-04-30T06:42:33.504ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 44037b3d-8f48-452c-9246-1cd50616319e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
52762025-04-30T06:42:33.504ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 44037b3d-8f48-452c-9246-1cd50616319e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
52772025-04-30T06:42:33.504ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 44037b3d-8f48-452c-9246-1cd50616319e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
52782025-04-30T06:42:33.505ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 44037b3d-8f48-452c-9246-1cd50616319e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
52792025-04-30T06:42:33.509ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 5720 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 44037b3d-8f48-452c-9246-1cd50616319e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
52802025-04-30T06:42:33.512ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "44037b3d-8f48-452c-9246-1cd50616319e", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52812025-04-30T06:42:33.512ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52822025-04-30T06:42:33.512ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
52832025-04-30T06:42:33.512ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
52842025-04-30T06:42:33.520ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82d3a79d-3aeb-44be-bc31-f0a9ab179c05", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52852025-04-30T06:42:33.520ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
52862025-04-30T06:42:33.526ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
52872025-04-30T06:42:33.528ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0ca6272-6c31-4937-a09f-b4ac098f5d20", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52882025-04-30T06:42:33.528ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
52892025-04-30T06:42:33.530ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eab141d-fa0d-4eae-9768-1a57b5fb73a8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52902025-04-30T06:42:33.531ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
52912025-04-30T06:42:33.533ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f8ab079-9eed-4cd4-bcd0-aba2cf436218", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52922025-04-30T06:42:33.533ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
52932025-04-30T06:42:33.535ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f4e5fbf-49c5-46ce-9aec-18f4bf3257ce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52942025-04-30T06:42:33.535ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
52952025-04-30T06:42:33.539ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2871f8b-4397-4af4-a91d-4a92d55d49b3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52962025-04-30T06:42:33.539ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
52972025-04-30T06:42:33.545ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac6785b-11d8-42ad-854e-b3bfc13c5825", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
52982025-04-30T06:42:33.545ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
52992025-04-30T06:42:33.551ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6e62368-24a4-4563-bcd1-b13666f122f3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
53002025-04-30T06:42:33.551ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
53012025-04-30T06:42:33.552ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53022025-04-30T06:42:33.552ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53032025-04-30T06:42:33.555ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53042025-04-30T06:42:33.556ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53052025-04-30T06:42:33.556ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53062025-04-30T06:42:33.556ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53072025-04-30T06:42:33.571ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99c7b247-8109-4415-b1f7-d8ff2ab1cb79", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:32 GMT"} })
53082025-04-30T06:42:33.586ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
53092025-04-30T06:42:33.588ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53102025-04-30T06:42:33.588ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53112025-04-30T06:42:33.588ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53122025-04-30T06:42:33.588ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
53132025-04-30T06:42:33.589ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53142025-04-30T06:42:33.592ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53152025-04-30T06:42:33.598ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53162025-04-30T06:42:33.598ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53172025-04-30T06:42:33.599ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
53182025-04-30T06:42:33.600ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53192025-04-30T06:42:33.603ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53202025-04-30T06:42:33.603ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
53212025-04-30T06:42:33.603ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53222025-04-30T06:42:33.606ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3663b0dd-9de5-4c68-97f8-2a3a8e6168b1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
53232025-04-30T06:42:33.606ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
53242025-04-30T06:42:33.617ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa2c41c1-c6e5-4247-9abf-3bddab05afde", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
53252025-04-30T06:42:33.617ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
53262025-04-30T06:42:33.623ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7157494-80ba-4841-8856-206a8ee9e779", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
53272025-04-30T06:42:33.624ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
53282025-04-30T06:42:33.626ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1f509dd-6a47-4f2f-bcb6-61b33f635a52", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
53292025-04-30T06:42:33.626ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
53302025-04-30T06:42:33.675ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "752f9698-5f89-4a01-be66-001543e0b466", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
53312025-04-30T06:42:33.675ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
53322025-04-30T06:42:33.678ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd36ca42-1120-4394-909c-4900aa295187", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
53332025-04-30T06:42:33.678ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
53342025-04-30T06:42:33.680ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c28595d6-b7bb-495a-9051-44f26077e67b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
53352025-04-30T06:42:33.680ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53362025-04-30T06:42:33.680ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53372025-04-30T06:42:33.680ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53382025-04-30T06:42:33.680ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53392025-04-30T06:42:33.682ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53402025-04-30T06:42:33.688ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53412025-04-30T06:42:33.701ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53422025-04-30T06:42:33.701ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53432025-04-30T06:42:33.701ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53442025-04-30T06:42:33.701ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53452025-04-30T06:42:33.701ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53462025-04-30T06:42:33.702ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53472025-04-30T06:42:33.707ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
53482025-04-30T06:42:33.709ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53492025-04-30T06:42:33.713ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53502025-04-30T06:42:33.713ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53512025-04-30T06:42:33.713ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53522025-04-30T06:42:33.714ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53532025-04-30T06:42:33.827ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
53542025-04-30T06:42:33.827ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
53552025-04-30T06:42:33.828ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53562025-04-30T06:42:33.828ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53572025-04-30T06:42:33.828ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53582025-04-30T06:42:33.828ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
53592025-04-30T06:42:33.828ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53602025-04-30T06:42:33.828ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:51132 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
53612025-04-30T06:42:33.830ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:51132 req_id = a0e58c91-8ccc-4aec-8294-6d0ee73cd3fa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53622025-04-30T06:42:33.830ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 678 local_addr = [::1]:62607 method = GET remote_addr = [::1]:51132 req_id = a0e58c91-8ccc-4aec-8294-6d0ee73cd3fa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53632025-04-30T06:42:33.831ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0e58c91-8ccc-4aec-8294-6d0ee73cd3fa", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
53642025-04-30T06:42:33.832ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53652025-04-30T06:42:33.832ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53662025-04-30T06:42:33.832ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
53672025-04-30T06:42:33.834ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53682025-04-30T06:42:33.834ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53692025-04-30T06:42:33.834ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53702025-04-30T06:42:33.834ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53712025-04-30T06:42:33.834ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53722025-04-30T06:42:33.835ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53732025-04-30T06:42:33.840ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53742025-04-30T06:42:33.840ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53752025-04-30T06:42:33.840ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
53762025-04-30T06:42:33.840ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53772025-04-30T06:42:33.842ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53782025-04-30T06:42:33.845ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53792025-04-30T06:42:33.846ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53802025-04-30T06:42:33.846ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53812025-04-30T06:42:33.846ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53822025-04-30T06:42:33.847ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53832025-04-30T06:42:33.847ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
53842025-04-30T06:42:33.847ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53852025-04-30T06:42:33.907ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53862025-04-30T06:42:33.907ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53872025-04-30T06:42:33.907ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
53882025-04-30T06:42:33.908ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
53892025-04-30T06:42:33.932ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
53902025-04-30T06:42:33.932ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
53912025-04-30T06:42:33.933ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
53922025-04-30T06:42:33.933ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
53932025-04-30T06:42:33.933ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 810.088609ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
53942025-04-30T06:42:33.933ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
53952025-04-30T06:42:33.933ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53962025-04-30T06:42:33.934ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53972025-04-30T06:42:33.934ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53982025-04-30T06:42:33.935ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53992025-04-30T06:42:33.935ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54002025-04-30T06:42:33.941ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
54012025-04-30T06:42:33.942ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
54022025-04-30T06:42:33.942ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54032025-04-30T06:42:33.943ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54042025-04-30T06:42:33.948ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
54052025-04-30T06:42:33.948ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54062025-04-30T06:42:33.948ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54072025-04-30T06:42:33.949ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54082025-04-30T06:42:33.957ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
54092025-04-30T06:42:33.957ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
54102025-04-30T06:42:33.958ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 88 4b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .K..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58011 req_id = 01c61301-2eb2-46f1-8592-77137e018dbe
54112025-04-30T06:42:33.959ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34891,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58011 req_id = 01c61301-2eb2-46f1-8592-77137e018dbe
54122025-04-30T06:42:33.959ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
54132025-04-30T06:42:33.959ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54142025-04-30T06:42:33.959ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58011 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 01c61301-2eb2-46f1-8592-77137e018dbe
54152025-04-30T06:42:33.959ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4218, tv_nsec: 758299396 } })
54162025-04-30T06:42:33.959ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
54172025-04-30T06:42:33.959ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0c b1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49807 req_id = eaf9ed07-28e3-450b-bf89-fba96b35e25c
54182025-04-30T06:42:33.959ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3249,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49807 req_id = eaf9ed07-28e3-450b-bf89-fba96b35e25c
54192025-04-30T06:42:33.960ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54202025-04-30T06:42:33.960ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49807 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = eaf9ed07-28e3-450b-bf89-fba96b35e25c
54212025-04-30T06:42:34.028ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
54222025-04-30T06:42:34.029ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54232025-04-30T06:42:34.029ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54242025-04-30T06:42:34.029ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54252025-04-30T06:42:34.030ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
54262025-04-30T06:42:34.031ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54272025-04-30T06:42:34.047ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
54282025-04-30T06:42:34.048ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3a 5d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c :]..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59986 req_id = 1226464e-d7aa-43b3-b498-43758b5fdbcb
54292025-04-30T06:42:34.048ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14941,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59986 req_id = 1226464e-d7aa-43b3-b498-43758b5fdbcb
54302025-04-30T06:42:34.048ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
54312025-04-30T06:42:34.048ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14941, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59986 req_id = 1226464e-d7aa-43b3-b498-43758b5fdbcb
54322025-04-30T06:42:34.048ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
54332025-04-30T06:42:34.048ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
54342025-04-30T06:42:34.048ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
54352025-04-30T06:42:34.051ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54362025-04-30T06:42:34.059ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54372025-04-30T06:42:34.059ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54382025-04-30T06:42:34.059ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
54392025-04-30T06:42:34.059ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54402025-04-30T06:42:34.060ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "331f9352-8d1b-4e61-b5ee-ff33143fa946", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54412025-04-30T06:42:34.060ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
54422025-04-30T06:42:34.061ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7ec93a6-75b9-4b4c-a322-25738f6a4442", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54432025-04-30T06:42:34.062ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
54442025-04-30T06:42:34.062ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54452025-04-30T06:42:34.062ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
54462025-04-30T06:42:34.063ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54472025-04-30T06:42:34.124ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38367845-44f1-43ac-87b1-d96f42936ece", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54482025-04-30T06:42:34.137ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
54492025-04-30T06:42:34.147ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "928458f1-95d7-4378-9923-8668087cfd57", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54502025-04-30T06:42:34.148ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
54512025-04-30T06:42:34.152ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
54522025-04-30T06:42:34.153ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63cfcd6f-51bd-4bd1-a5e3-da6606c306cb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54532025-04-30T06:42:34.153ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
54542025-04-30T06:42:34.155ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82154dcd-f2c5-4ad8-ad9d-2213b96a3669", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54552025-04-30T06:42:34.155ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
54562025-04-30T06:42:34.157ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "319e6300-95d0-4c6a-9a02-4e4a43766475", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54572025-04-30T06:42:34.157ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
54582025-04-30T06:42:34.159ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed61585-9169-44d1-879b-a50b92fd7753", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54592025-04-30T06:42:34.159ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
54602025-04-30T06:42:34.160ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c7da67f-d159-4fc6-8348-0451076e9c49", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54612025-04-30T06:42:34.160ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
54622025-04-30T06:42:34.166ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "325a0265-4754-4b1b-b23f-cbab047d3406", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54632025-04-30T06:42:34.166ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
54642025-04-30T06:42:34.167ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b0320ba-5f3c-47c6-842a-daa3299e824d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54652025-04-30T06:42:34.167ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
54662025-04-30T06:42:34.168ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fd99aa0-fb71-4c2f-9975-9921d286232c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54672025-04-30T06:42:34.168ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
54682025-04-30T06:42:34.171ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54c5b010-a956-4d79-823a-fa690b8eb47e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54692025-04-30T06:42:34.171ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
54702025-04-30T06:42:34.175ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6565897f-236d-42b6-a5c0-381920b102c2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54712025-04-30T06:42:34.175ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
54722025-04-30T06:42:34.179ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "976b4568-3869-462d-8984-e254f7956244", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54732025-04-30T06:42:34.180ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
54742025-04-30T06:42:34.184ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f973db5-6be7-4544-befd-80de953f349c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54752025-04-30T06:42:34.184ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
54762025-04-30T06:42:34.188ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d93be2b-4da7-4a5f-b37b-7325ef145669", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54772025-04-30T06:42:34.188ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
54782025-04-30T06:42:34.192ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12a68249-7ba3-4133-8e3a-a839a8cce4d8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54792025-04-30T06:42:34.193ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
54802025-04-30T06:42:34.198ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d082d876-ae8e-4038-bb37-21bc9cfc2c4b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54812025-04-30T06:42:34.198ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
54822025-04-30T06:42:34.203ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bebbf63e-c756-41ce-9af2-beac0f72b217", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54832025-04-30T06:42:34.203ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
54842025-04-30T06:42:34.203ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54852025-04-30T06:42:34.203ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54862025-04-30T06:42:34.203ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54872025-04-30T06:42:34.203ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
54882025-04-30T06:42:34.204ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54892025-04-30T06:42:34.214ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54902025-04-30T06:42:34.220ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54912025-04-30T06:42:34.220ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54922025-04-30T06:42:34.220ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
54932025-04-30T06:42:34.221ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54942025-04-30T06:42:34.222ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d958335-4e32-43d6-9522-4b6bfe9b296f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
54952025-04-30T06:42:34.222ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
54962025-04-30T06:42:34.228ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54972025-04-30T06:42:34.228ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
54982025-04-30T06:42:34.229ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54992025-04-30T06:42:34.230ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c3ca6f3-05da-4d5c-9573-1e6f4f1f648e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55002025-04-30T06:42:34.230ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
55012025-04-30T06:42:34.231ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b63ba7e-5e9a-480a-9248-78cd95246d7c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55022025-04-30T06:42:34.231ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
55032025-04-30T06:42:34.265ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bcbddc6-8d62-490a-8f4f-6624158bf85e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55042025-04-30T06:42:34.265ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
55052025-04-30T06:42:34.269ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e317e8e0-4133-4bec-a0aa-ee5a7ed0e8ef", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55062025-04-30T06:42:34.269ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
55072025-04-30T06:42:34.273ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "099ba871-c848-4808-b2be-9d123efe1be1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55082025-04-30T06:42:34.297ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
55092025-04-30T06:42:34.309ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b435207-a202-4a7f-a95e-2b71c78e9862", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55102025-04-30T06:42:34.309ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
55112025-04-30T06:42:34.327ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8a9b16c-16f1-47af-9853-c74297c8396c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55122025-04-30T06:42:34.327ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
55132025-04-30T06:42:34.335ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
55142025-04-30T06:42:34.337ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef3e1569-95ba-4dcd-b2ff-f58a17138c8b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55152025-04-30T06:42:34.339ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
55162025-04-30T06:42:34.341ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b139e88-971c-421e-ac03-0e0939533738", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55172025-04-30T06:42:34.341ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
55182025-04-30T06:42:34.343ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ab6e4f7-9886-4f49-b4e9-782b4a3e5f2b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55192025-04-30T06:42:34.343ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
55202025-04-30T06:42:34.346ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a29a222-57b8-4a0f-8770-4eb2e038e9f9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55212025-04-30T06:42:34.348ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
55222025-04-30T06:42:34.349ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2b1f30a-4877-4aa4-91db-180052865464", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55232025-04-30T06:42:34.349ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
55242025-04-30T06:42:34.350ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "741415c5-c18f-4d5a-ba9a-30457d793dd8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55252025-04-30T06:42:34.351ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
55262025-04-30T06:42:34.352ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ab43a1c-86db-4a2e-a50f-17b68fdf7fb4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55272025-04-30T06:42:34.353ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
55282025-04-30T06:42:34.355ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c1cdb6b-34af-4142-a3ab-40098aebfee4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55292025-04-30T06:42:34.355ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
55302025-04-30T06:42:34.357ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92b9ca82-2160-44d7-8ee3-14e57aa10d3b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55312025-04-30T06:42:34.358ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
55322025-04-30T06:42:34.359ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d14efa58-b6b9-472b-88f6-f89255bf8a7c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55332025-04-30T06:42:34.361ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
55342025-04-30T06:42:34.365ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e95527df-6c8b-45ab-8079-c68701db56a5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55352025-04-30T06:42:34.365ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
55362025-04-30T06:42:34.376ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21adc330-b80b-4b1e-b632-1cb9d5fe87ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55372025-04-30T06:42:34.376ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
55382025-04-30T06:42:34.378ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "752f76f9-7479-4dac-8ad8-f5367f73d452", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55392025-04-30T06:42:34.379ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
55402025-04-30T06:42:34.381ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d377f8f-e5aa-4726-9150-b4c93a16b94c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55412025-04-30T06:42:34.381ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
55422025-04-30T06:42:34.384ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "503abf78-dcd4-4aab-add5-fb9649e29712", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55432025-04-30T06:42:34.384ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
55442025-04-30T06:42:34.385ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd211e6a-5703-497c-9c8c-98d9958922e5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55452025-04-30T06:42:34.385ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
55462025-04-30T06:42:34.386ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55472025-04-30T06:42:34.386ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55482025-04-30T06:42:34.386ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55492025-04-30T06:42:34.386ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
55502025-04-30T06:42:34.390ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55512025-04-30T06:42:34.393ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b0424ac-fc07-48a0-80e8-c8a17838e357", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55522025-04-30T06:42:34.393ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55532025-04-30T06:42:34.393ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55542025-04-30T06:42:34.393ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55552025-04-30T06:42:34.393ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55562025-04-30T06:42:34.393ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55572025-04-30T06:42:34.394ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55582025-04-30T06:42:34.407ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55592025-04-30T06:42:34.407ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55602025-04-30T06:42:34.407ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
55612025-04-30T06:42:34.408ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55622025-04-30T06:42:34.410ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55632025-04-30T06:42:34.422ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55642025-04-30T06:42:34.423ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55652025-04-30T06:42:34.423ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55662025-04-30T06:42:34.423ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55672025-04-30T06:42:34.423ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55682025-04-30T06:42:34.424ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55692025-04-30T06:42:34.430ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55702025-04-30T06:42:34.430ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
55712025-04-30T06:42:34.431ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55722025-04-30T06:42:34.432ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55732025-04-30T06:42:34.452ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55742025-04-30T06:42:34.453ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55752025-04-30T06:42:34.454ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55762025-04-30T06:42:34.455ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55772025-04-30T06:42:34.564ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
55782025-04-30T06:42:34.564ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
55792025-04-30T06:42:34.575ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:39368 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55802025-04-30T06:42:34.575ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55812025-04-30T06:42:34.575ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55822025-04-30T06:42:34.576ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:39368 req_id = 5fb935e5-cdfe-4750-b4d7-1b14e382324b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55832025-04-30T06:42:34.579ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 2703 local_addr = [::1]:62607 method = GET remote_addr = [::1]:39368 req_id = 5fb935e5-cdfe-4750-b4d7-1b14e382324b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55842025-04-30T06:42:34.587ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55852025-04-30T06:42:34.595ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55862025-04-30T06:42:34.595ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55872025-04-30T06:42:34.595ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55882025-04-30T06:42:34.602ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fb935e5-cdfe-4750-b4d7-1b14e382324b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:33 GMT"} })
55892025-04-30T06:42:34.611ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55902025-04-30T06:42:34.612ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55912025-04-30T06:42:34.612ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
55922025-04-30T06:42:34.614ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55932025-04-30T06:42:34.614ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55942025-04-30T06:42:34.614ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55952025-04-30T06:42:34.614ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55962025-04-30T06:42:34.615ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55972025-04-30T06:42:34.619ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55982025-04-30T06:42:34.626ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55992025-04-30T06:42:34.626ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56002025-04-30T06:42:34.626ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56012025-04-30T06:42:34.628ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56022025-04-30T06:42:34.673ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
56032025-04-30T06:42:34.691ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
56042025-04-30T06:42:34.691ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
56052025-04-30T06:42:34.691ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 758.026689ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
56062025-04-30T06:42:34.691ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
56072025-04-30T06:42:34.691ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
56082025-04-30T06:42:34.691ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56092025-04-30T06:42:34.691ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56102025-04-30T06:42:34.691ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56112025-04-30T06:42:34.692ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56122025-04-30T06:42:34.695ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56132025-04-30T06:42:34.706ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
56142025-04-30T06:42:34.707ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56152025-04-30T06:42:34.707ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56162025-04-30T06:42:34.708ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56172025-04-30T06:42:34.713ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
56182025-04-30T06:42:34.713ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
56192025-04-30T06:42:34.715ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 06 2d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .-..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64368 req_id = 2be2d89d-161d-46a1-8200-d4319b046493
56202025-04-30T06:42:34.715ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1581,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64368 req_id = 2be2d89d-161d-46a1-8200-d4319b046493
56212025-04-30T06:42:34.715ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
56222025-04-30T06:42:34.715ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56232025-04-30T06:42:34.716ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64368 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2be2d89d-161d-46a1-8200-d4319b046493
56242025-04-30T06:42:34.716ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4219, tv_nsec: 515395190 } })
56252025-04-30T06:42:34.716ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
56262025-04-30T06:42:34.717ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 17 05 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54585 req_id = e18a8c18-f6dd-4f9c-95e8-2e18e102c505
56272025-04-30T06:42:34.717ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5893,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54585 req_id = e18a8c18-f6dd-4f9c-95e8-2e18e102c505
56282025-04-30T06:42:34.717ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56292025-04-30T06:42:34.717ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54585 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e18a8c18-f6dd-4f9c-95e8-2e18e102c505
56302025-04-30T06:42:34.783ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
56312025-04-30T06:42:34.784ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56322025-04-30T06:42:34.784ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56332025-04-30T06:42:34.784ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56342025-04-30T06:42:34.784ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
56352025-04-30T06:42:34.784ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56362025-04-30T06:42:34.785ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 99 7f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57154 req_id = 6aeaf7ae-3343-4779-bbb2-17b4694df13d
56372025-04-30T06:42:34.785ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39295,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57154 req_id = 6aeaf7ae-3343-4779-bbb2-17b4694df13d
56382025-04-30T06:42:34.785ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
56392025-04-30T06:42:34.785ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39295, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57154 req_id = 6aeaf7ae-3343-4779-bbb2-17b4694df13d
56402025-04-30T06:42:34.785ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
56412025-04-30T06:42:34.785ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
56422025-04-30T06:42:34.785ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
56432025-04-30T06:42:34.787ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "63176ee0-8425-41b1-8115-a368c725c647", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56442025-04-30T06:42:34.787ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
56452025-04-30T06:42:34.787ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56462025-04-30T06:42:34.792ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56472025-04-30T06:42:34.792ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56482025-04-30T06:42:34.792ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
56492025-04-30T06:42:34.792ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56502025-04-30T06:42:34.794ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11db8c17-5aa3-4646-a422-1917d0d3516f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56512025-04-30T06:42:34.794ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
56522025-04-30T06:42:34.795ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56532025-04-30T06:42:34.795ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
56542025-04-30T06:42:34.795ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56552025-04-30T06:42:34.797ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3bae48d-246a-4a2c-8c42-3df9c5b28c46", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56562025-04-30T06:42:34.797ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
56572025-04-30T06:42:34.799ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4467dd47-c01d-444c-9a11-1d9112ba20d8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56582025-04-30T06:42:34.799ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
56592025-04-30T06:42:34.802ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed342a29-62ba-4956-99bf-8b642ac54f62", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56602025-04-30T06:42:34.802ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
56612025-04-30T06:42:34.804ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3a1d2a7-5640-4486-a987-4163be6b06c7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56622025-04-30T06:42:34.805ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
56632025-04-30T06:42:34.808ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03cc4fd9-ee38-4495-b836-a4f9db0444bd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56642025-04-30T06:42:34.809ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
56652025-04-30T06:42:34.813ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e64aee3-92bd-4edb-be3a-c497730029f8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56662025-04-30T06:42:34.813ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
56672025-04-30T06:42:34.814ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f3c1c36-748e-4240-8285-fe4a20605aa5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56682025-04-30T06:42:34.814ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
56692025-04-30T06:42:34.815ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fc94133-2de6-4d3d-b408-5f5a28d91df3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56702025-04-30T06:42:34.816ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
56712025-04-30T06:42:34.817ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79231241-09f2-465b-9e05-1d6ede9e276d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56722025-04-30T06:42:34.817ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
56732025-04-30T06:42:34.818ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c64b41be-651c-4978-9d45-c771fa71ae86", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56742025-04-30T06:42:34.818ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
56752025-04-30T06:42:34.820ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf6e16c0-7526-46bc-9004-e36143496e32", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56762025-04-30T06:42:34.820ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
56772025-04-30T06:42:34.824ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
56782025-04-30T06:42:34.824ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5d5f537-32fe-4022-8c09-f3928571af45", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56792025-04-30T06:42:34.824ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
56802025-04-30T06:42:34.827ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "794e8b8a-ac81-4181-a33a-a8c6f17028a6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56812025-04-30T06:42:34.827ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
56822025-04-30T06:42:34.829ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4897962-5985-4f56-81c5-73ff005e123a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56832025-04-30T06:42:34.829ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
56842025-04-30T06:42:34.833ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "012d5d3d-a59b-4848-a41f-50e1ede0c6f5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56852025-04-30T06:42:34.833ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
56862025-04-30T06:42:34.835ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "972bd80f-abab-4427-bf3e-c3824d9545df", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56872025-04-30T06:42:34.835ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
56882025-04-30T06:42:34.837ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b7556b0-cb4c-46fa-81a7-c68a83de92ad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56892025-04-30T06:42:34.837ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
56902025-04-30T06:42:34.839ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e98e5038-2cf9-4152-b505-3a954f8074f1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56912025-04-30T06:42:34.839ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
56922025-04-30T06:42:34.841ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f52523de-c5e7-4d3f-a70c-dc89a0a4cf87", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56932025-04-30T06:42:34.841ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
56942025-04-30T06:42:34.843ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bba5ddd-f1b9-402f-b6b8-87eb637e2036", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56952025-04-30T06:42:34.843ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
56962025-04-30T06:42:34.844ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65e42897-62bb-47f4-999c-c198a0134af1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56972025-04-30T06:42:34.844ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
56982025-04-30T06:42:34.845ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe16bde6-92a2-4dbe-bfa5-60b56c719401", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
56992025-04-30T06:42:34.846ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
57002025-04-30T06:42:34.848ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b48febf0-0075-4d1e-a17d-4ff2698d31b3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57012025-04-30T06:42:34.848ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
57022025-04-30T06:42:34.850ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "540d7f40-2834-4b83-a885-896ee3ee0249", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57032025-04-30T06:42:34.850ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
57042025-04-30T06:42:34.853ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2fd7f77-c512-4fb8-b21f-27c033b1687c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57052025-04-30T06:42:34.853ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
57062025-04-30T06:42:34.855ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8bdb46d-0485-4c43-840b-6c5156c62103", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57072025-04-30T06:42:34.856ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
57082025-04-30T06:42:34.857ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7bfbafd-bc99-4893-9606-ae1fae4282b6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57092025-04-30T06:42:34.857ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
57102025-04-30T06:42:34.860ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b98d3d7-792a-41c0-89c8-666dc510eb25", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57112025-04-30T06:42:34.860ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
57122025-04-30T06:42:34.861ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "813979d0-6e7a-4371-9a7c-49be2d336797", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57132025-04-30T06:42:34.861ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
57142025-04-30T06:42:34.862ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dd43b35-5e11-460d-bbfc-b8c7251bc6f8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57152025-04-30T06:42:34.862ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
57162025-04-30T06:42:34.863ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f3cdf43-9578-4dc5-974c-22cf06ad1a76", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57172025-04-30T06:42:34.864ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
57182025-04-30T06:42:34.865ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ee58e75-230e-4cf5-b89b-c5a1503611e3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57192025-04-30T06:42:34.865ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
57202025-04-30T06:42:34.866ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2611aebd-567a-4fe8-aa0a-bef2caa2b729", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57212025-04-30T06:42:34.866ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
57222025-04-30T06:42:34.868ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55e3323f-38ee-4784-8481-19236d11c131", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57232025-04-30T06:42:34.868ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
57242025-04-30T06:42:34.870ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8488225b-d7ab-40dc-80ef-a05774b3d78c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57252025-04-30T06:42:34.870ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
57262025-04-30T06:42:34.872ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65c7a47f-0f24-4fd6-8153-e3194c94334c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57272025-04-30T06:42:34.872ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
57282025-04-30T06:42:34.875ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58523ecf-cacc-4a8c-bf7f-38cd51e670e8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57292025-04-30T06:42:34.875ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
57302025-04-30T06:42:34.875ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57312025-04-30T06:42:34.875ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57322025-04-30T06:42:34.875ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57332025-04-30T06:42:34.875ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
57342025-04-30T06:42:34.877ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57352025-04-30T06:42:34.880ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e5e2ad2-5b7c-40a3-9e6a-4b99955ae015", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57362025-04-30T06:42:34.880ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
57372025-04-30T06:42:34.880ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57382025-04-30T06:42:34.885ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57392025-04-30T06:42:34.885ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57402025-04-30T06:42:34.885ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
57412025-04-30T06:42:34.886ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57422025-04-30T06:42:34.889ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57432025-04-30T06:42:34.889ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
57442025-04-30T06:42:34.889ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57452025-04-30T06:42:34.892ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5218b6-1a29-4b2b-bef3-7361a3dd737a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57462025-04-30T06:42:34.892ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
57472025-04-30T06:42:34.897ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d9c2a9e-572e-45b0-95c1-e306d95189ad", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57482025-04-30T06:42:34.898ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
57492025-04-30T06:42:34.901ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e513a782-0b83-434b-a3dd-63d3c27668be", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57502025-04-30T06:42:34.903ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
57512025-04-30T06:42:34.903ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ee01e65-f3df-4718-9262-d3ef14b95af1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57522025-04-30T06:42:34.903ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
57532025-04-30T06:42:34.903ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57542025-04-30T06:42:34.903ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57552025-04-30T06:42:34.903ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
57562025-04-30T06:42:34.903ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
57572025-04-30T06:42:34.907ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2111b11-07fb-4dfc-9417-6a6de268b348", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57582025-04-30T06:42:34.907ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57592025-04-30T06:42:34.907ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57602025-04-30T06:42:34.907ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57612025-04-30T06:42:34.907ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57622025-04-30T06:42:34.910ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57632025-04-30T06:42:34.913ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
57642025-04-30T06:42:34.913ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
57652025-04-30T06:42:34.915ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57662025-04-30T06:42:34.921ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57672025-04-30T06:42:34.921ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57682025-04-30T06:42:34.921ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57692025-04-30T06:42:34.921ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57702025-04-30T06:42:34.922ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57712025-04-30T06:42:34.924ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57722025-04-30T06:42:34.925ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
57732025-04-30T06:42:34.925ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57742025-04-30T06:42:34.930ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57752025-04-30T06:42:34.936ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57762025-04-30T06:42:34.936ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57772025-04-30T06:42:34.936ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57782025-04-30T06:42:34.942ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57792025-04-30T06:42:35.035ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
57802025-04-30T06:42:35.035ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
57812025-04-30T06:42:35.036ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:52002 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
57822025-04-30T06:42:35.036ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
57832025-04-30T06:42:35.036ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:52002 req_id = 85c3dddd-20a1-46ab-8c6d-ffe9485526f1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57842025-04-30T06:42:35.037ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 680 local_addr = [::1]:62607 method = GET remote_addr = [::1]:52002 req_id = 85c3dddd-20a1-46ab-8c6d-ffe9485526f1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57852025-04-30T06:42:35.037ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "85c3dddd-20a1-46ab-8c6d-ffe9485526f1", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
57862025-04-30T06:42:35.040ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57872025-04-30T06:42:35.040ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57882025-04-30T06:42:35.040ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
57892025-04-30T06:42:35.041ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57902025-04-30T06:42:35.041ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57912025-04-30T06:42:35.041ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57922025-04-30T06:42:35.041ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57932025-04-30T06:42:35.042ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57942025-04-30T06:42:35.060ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
57952025-04-30T06:42:35.067ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57962025-04-30T06:42:35.076ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57972025-04-30T06:42:35.076ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57982025-04-30T06:42:35.076ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57992025-04-30T06:42:35.077ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58002025-04-30T06:42:35.114ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58012025-04-30T06:42:35.114ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58022025-04-30T06:42:35.114ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58032025-04-30T06:42:35.114ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
58042025-04-30T06:42:35.115ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58052025-04-30T06:42:35.119ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58062025-04-30T06:42:35.124ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58072025-04-30T06:42:35.125ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58082025-04-30T06:42:35.125ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
58092025-04-30T06:42:35.126ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58102025-04-30T06:42:35.142ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58112025-04-30T06:42:35.142ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
58122025-04-30T06:42:35.143ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58132025-04-30T06:42:35.143ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
58142025-04-30T06:42:35.144ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
58152025-04-30T06:42:35.145ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 452.679575ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
58162025-04-30T06:42:35.145ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
58172025-04-30T06:42:35.145ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
58182025-04-30T06:42:35.145ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58192025-04-30T06:42:35.145ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58202025-04-30T06:42:35.145ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58212025-04-30T06:42:35.145ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58222025-04-30T06:42:35.148ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58232025-04-30T06:42:35.153ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
58242025-04-30T06:42:35.153ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58252025-04-30T06:42:35.153ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58262025-04-30T06:42:35.154ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58272025-04-30T06:42:35.169ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
58282025-04-30T06:42:35.169ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
58292025-04-30T06:42:35.169ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 42 0f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 B............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49589 req_id = e6ed1003-76bf-408e-9994-f0406d02fdb7
58302025-04-30T06:42:35.170ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16911,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49589 req_id = e6ed1003-76bf-408e-9994-f0406d02fdb7
58312025-04-30T06:42:35.170ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
58322025-04-30T06:42:35.170ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58332025-04-30T06:42:35.170ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49589 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e6ed1003-76bf-408e-9994-f0406d02fdb7
58342025-04-30T06:42:35.171ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4219, tv_nsec: 970047435 } })
58352025-04-30T06:42:35.171ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
58362025-04-30T06:42:35.171ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3f b9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ?...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58176 req_id = 99fd2571-f7ba-4e6f-a5a3-6f2d5129c7a3
58372025-04-30T06:42:35.171ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16313,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58176 req_id = 99fd2571-f7ba-4e6f-a5a3-6f2d5129c7a3
58382025-04-30T06:42:35.172ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58392025-04-30T06:42:35.172ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58176 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 99fd2571-f7ba-4e6f-a5a3-6f2d5129c7a3
58402025-04-30T06:42:35.237ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
58412025-04-30T06:42:35.238ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e6 5c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .\\..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53750 req_id = d14f9d88-0dd6-4afa-b79c-386f593a19c6
58422025-04-30T06:42:35.238ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58972,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53750 req_id = d14f9d88-0dd6-4afa-b79c-386f593a19c6
58432025-04-30T06:42:35.238ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
58442025-04-30T06:42:35.238ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58972, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53750 req_id = d14f9d88-0dd6-4afa-b79c-386f593a19c6
58452025-04-30T06:42:35.239ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
58462025-04-30T06:42:35.239ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
58472025-04-30T06:42:35.239ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
58482025-04-30T06:42:35.240ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0a64c3c-7b68-4bb3-9c6b-84a1e6c28a95", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58492025-04-30T06:42:35.241ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
58502025-04-30T06:42:35.242ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e5fafb5-bf1a-4fca-9474-4f0300cf80ed", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58512025-04-30T06:42:35.244ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
58522025-04-30T06:42:35.251ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41eefd24-4e32-4dd1-a892-ad4ddff97bd9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58532025-04-30T06:42:35.258ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
58542025-04-30T06:42:35.264ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
58552025-04-30T06:42:35.266ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27822300-4831-44eb-9ecb-ff02ae2046b7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58562025-04-30T06:42:35.266ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
58572025-04-30T06:42:35.269ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a553589-8218-439b-b543-7551917f8208", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58582025-04-30T06:42:35.269ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
58592025-04-30T06:42:35.271ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2afc1868-0807-4877-8de0-e462a1606873", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58602025-04-30T06:42:35.271ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
58612025-04-30T06:42:35.273ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5c2c0aa-dcb3-44ca-8607-ec9a7f9869ea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58622025-04-30T06:42:35.273ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
58632025-04-30T06:42:35.275ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66a2c8bc-ef73-45b7-b4a5-9ed5a2d6d433", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58642025-04-30T06:42:35.275ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
58652025-04-30T06:42:35.276ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e342405-c838-430c-b559-0852811cda88", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58662025-04-30T06:42:35.276ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
58672025-04-30T06:42:35.277ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "514111dc-2368-4e13-a28d-ee06d1bdbfda", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58682025-04-30T06:42:35.278ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
58692025-04-30T06:42:35.279ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc1df4e5-a083-4de5-af01-f4704e69b285", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58702025-04-30T06:42:35.279ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
58712025-04-30T06:42:35.280ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "438bce68-e395-4ed2-a1f5-aed10bcc7bd3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58722025-04-30T06:42:35.280ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
58732025-04-30T06:42:35.281ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62554d6b-f6de-4f36-9c1c-27eb4e880277", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58742025-04-30T06:42:35.281ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
58752025-04-30T06:42:35.283ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae12889f-7869-401c-bc7e-222b28f6190f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58762025-04-30T06:42:35.284ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
58772025-04-30T06:42:35.285ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2ceccd1-2608-4155-9b9f-75771b053693", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58782025-04-30T06:42:35.286ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
58792025-04-30T06:42:35.287ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06dcdf61-b2c0-4135-848e-6ece735a64b8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58802025-04-30T06:42:35.288ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
58812025-04-30T06:42:35.289ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "79977a7b-62d8-4277-97b6-015815d04a0a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58822025-04-30T06:42:35.290ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
58832025-04-30T06:42:35.292ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06d43256-f4d8-44cc-b524-1bb200be6380", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58842025-04-30T06:42:35.292ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
58852025-04-30T06:42:35.294ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e59a4329-368e-4573-82dd-ade607511b78", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58862025-04-30T06:42:35.294ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
58872025-04-30T06:42:35.295ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b533fd6a-1456-440b-8a84-ebba5a16dd29", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58882025-04-30T06:42:35.295ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
58892025-04-30T06:42:35.297ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ccca4ed-e4d6-454e-a8f7-8fd66e610f61", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58902025-04-30T06:42:35.297ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
58912025-04-30T06:42:35.298ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10c9cb88-488f-41f5-aeb5-56d9cbe037ac", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58922025-04-30T06:42:35.298ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
58932025-04-30T06:42:35.299ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e33f39f6-ce49-48f1-8500-66124419711d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58942025-04-30T06:42:35.299ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
58952025-04-30T06:42:35.301ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74fc718f-7bb4-4817-98d6-656a0e2a540a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58962025-04-30T06:42:35.301ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
58972025-04-30T06:42:35.303ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cb6561a-f29b-4cd0-aa53-67566685745f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
58982025-04-30T06:42:35.303ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
58992025-04-30T06:42:35.306ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ede3be4-5172-449e-994a-da7758dcd62c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59002025-04-30T06:42:35.306ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
59012025-04-30T06:42:35.308ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6360699d-7b7b-4111-9ded-767d75a5062c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59022025-04-30T06:42:35.308ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
59032025-04-30T06:42:35.310ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eae19041-9ad7-48ee-90ef-4835c0c38d59", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59042025-04-30T06:42:35.310ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
59052025-04-30T06:42:35.312ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e330dfb5-593a-4a87-8e02-a5acfeba6356", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59062025-04-30T06:42:35.313ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
59072025-04-30T06:42:35.315ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59082025-04-30T06:42:35.315ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59092025-04-30T06:42:35.315ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59102025-04-30T06:42:35.315ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
59112025-04-30T06:42:35.316ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59122025-04-30T06:42:35.317ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c753683-b9aa-49f1-9ff4-55cf3e548a46", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59132025-04-30T06:42:35.317ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
59142025-04-30T06:42:35.319ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab81c98d-fbf9-487d-a927-c891046455d1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59152025-04-30T06:42:35.319ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
59162025-04-30T06:42:35.320ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59172025-04-30T06:42:35.328ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59182025-04-30T06:42:35.328ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59192025-04-30T06:42:35.328ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
59202025-04-30T06:42:35.329ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59212025-04-30T06:42:35.332ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0359ba4b-0f1d-4783-af37-758431ba51e8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59222025-04-30T06:42:35.332ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
59232025-04-30T06:42:35.332ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59242025-04-30T06:42:35.332ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
59252025-04-30T06:42:35.333ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59262025-04-30T06:42:35.335ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a696fffb-fe3c-42af-87e9-2e8fde179553", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59272025-04-30T06:42:35.335ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
59282025-04-30T06:42:35.336ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca18dd56-38f3-44b1-a313-9e3858f09607", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59292025-04-30T06:42:35.336ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
59302025-04-30T06:42:35.337ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f78672f5-3850-457a-946c-dde93e2757ce", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59312025-04-30T06:42:35.337ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
59322025-04-30T06:42:35.339ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb773800-a32c-4602-8efe-ca86857f9968", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59332025-04-30T06:42:35.339ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
59342025-04-30T06:42:35.343ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b844c2ef-858b-459f-a5cb-3100ea1a6a60", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59352025-04-30T06:42:35.343ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
59362025-04-30T06:42:35.347ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee5826f8-a831-405b-b3f8-7bb49a955b71", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59372025-04-30T06:42:35.348ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
59382025-04-30T06:42:35.352ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1c2d1fe-a61f-4cdd-a392-44e4e2520d0c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59392025-04-30T06:42:35.352ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
59402025-04-30T06:42:35.356ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6245b90-9bd5-4a13-941c-c7d15f04f3bd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59412025-04-30T06:42:35.356ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
59422025-04-30T06:42:35.359ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6ac1f1f-3b14-4e02-bd03-e1e6409ce616", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59432025-04-30T06:42:35.360ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
59442025-04-30T06:42:35.360ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8cb7270-ddd3-4a36-ae9f-7f2de45f506b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59452025-04-30T06:42:35.360ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
59462025-04-30T06:42:35.361ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d37a8960-9bec-4742-88ae-36bdb32188ab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59472025-04-30T06:42:35.361ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
59482025-04-30T06:42:35.362ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8861a8ae-e82d-4ccf-823e-4a1713676c83", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59492025-04-30T06:42:35.362ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
59502025-04-30T06:42:35.362ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "881d0e11-3e99-4bbc-8d1e-7e3f8978870b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59512025-04-30T06:42:35.362ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59522025-04-30T06:42:35.363ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59532025-04-30T06:42:35.363ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59542025-04-30T06:42:35.363ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59552025-04-30T06:42:35.363ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59562025-04-30T06:42:35.369ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59572025-04-30T06:42:35.377ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59582025-04-30T06:42:35.377ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59592025-04-30T06:42:35.377ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59602025-04-30T06:42:35.377ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59612025-04-30T06:42:35.377ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59622025-04-30T06:42:35.379ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59632025-04-30T06:42:35.379ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
59642025-04-30T06:42:35.382ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59652025-04-30T06:42:35.387ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59662025-04-30T06:42:35.387ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59672025-04-30T06:42:35.387ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59682025-04-30T06:42:35.389ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59692025-04-30T06:42:35.460ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
59702025-04-30T06:42:35.461ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
59712025-04-30T06:42:35.461ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59722025-04-30T06:42:35.461ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59732025-04-30T06:42:35.461ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59742025-04-30T06:42:35.461ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
59752025-04-30T06:42:35.463ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59762025-04-30T06:42:35.463ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:59892 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
59772025-04-30T06:42:35.464ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:59892 req_id = 0353de7d-ba8d-49ba-ad39-c9e10e3d2e14 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59782025-04-30T06:42:35.464ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 565 local_addr = [::1]:62607 method = GET remote_addr = [::1]:59892 req_id = 0353de7d-ba8d-49ba-ad39-c9e10e3d2e14 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59792025-04-30T06:42:35.465ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0353de7d-ba8d-49ba-ad39-c9e10e3d2e14", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
59802025-04-30T06:42:35.468ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59812025-04-30T06:42:35.468ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59822025-04-30T06:42:35.468ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
59832025-04-30T06:42:35.470ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59842025-04-30T06:42:35.471ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59852025-04-30T06:42:35.471ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59862025-04-30T06:42:35.471ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59872025-04-30T06:42:35.471ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59882025-04-30T06:42:35.472ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59892025-04-30T06:42:35.476ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59902025-04-30T06:42:35.476ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59912025-04-30T06:42:35.476ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
59922025-04-30T06:42:35.477ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59932025-04-30T06:42:35.479ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59942025-04-30T06:42:35.479ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
59952025-04-30T06:42:35.480ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59962025-04-30T06:42:35.480ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59972025-04-30T06:42:35.484ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59982025-04-30T06:42:35.484ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59992025-04-30T06:42:35.484ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60002025-04-30T06:42:35.485ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60012025-04-30T06:42:35.489ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
60022025-04-30T06:42:35.489ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60032025-04-30T06:42:35.490ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ae357156-fb4c-4f6e-99e6-a3badbf3bb93 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60042025-04-30T06:42:35.490ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ae357156-fb4c-4f6e-99e6-a3badbf3bb93 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60052025-04-30T06:42:35.490ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ae357156-fb4c-4f6e-99e6-a3badbf3bb93 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60062025-04-30T06:42:35.491ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ae357156-fb4c-4f6e-99e6-a3badbf3bb93 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60072025-04-30T06:42:35.494ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 4875 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = ae357156-fb4c-4f6e-99e6-a3badbf3bb93 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60082025-04-30T06:42:35.496ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae357156-fb4c-4f6e-99e6-a3badbf3bb93", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
60092025-04-30T06:42:35.496ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
60102025-04-30T06:42:35.497ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9da6031b-4b94-4c11-841b-b431a8dd346f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
60112025-04-30T06:42:35.498ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9da6031b-4b94-4c11-841b-b431a8dd346f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
60122025-04-30T06:42:35.498ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9da6031b-4b94-4c11-841b-b431a8dd346f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
60132025-04-30T06:42:35.499ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9da6031b-4b94-4c11-841b-b431a8dd346f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
60142025-04-30T06:42:35.503ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 5238 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9da6031b-4b94-4c11-841b-b431a8dd346f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
60152025-04-30T06:42:35.503ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "9da6031b-4b94-4c11-841b-b431a8dd346f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:34 GMT"} })
60162025-04-30T06:42:35.505ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60172025-04-30T06:42:35.505ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
60182025-04-30T06:42:35.505ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
60192025-04-30T06:42:35.529ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
60202025-04-30T06:42:35.530ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
60212025-04-30T06:42:35.531ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
60222025-04-30T06:42:35.531ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 386.685466ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
60232025-04-30T06:42:35.531ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
60242025-04-30T06:42:35.531ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60252025-04-30T06:42:35.531ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60262025-04-30T06:42:35.531ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60272025-04-30T06:42:35.531ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60282025-04-30T06:42:35.532ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60292025-04-30T06:42:35.536ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60302025-04-30T06:42:35.543ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60312025-04-30T06:42:35.543ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60322025-04-30T06:42:35.543ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60332025-04-30T06:42:35.543ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60342025-04-30T06:42:35.548ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
60352025-04-30T06:42:35.548ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
60362025-04-30T06:42:35.550ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3f 59 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ?Y..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62061 req_id = 39a9b09c-2c49-4a80-98be-5f11dafc0053
60372025-04-30T06:42:35.550ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16217,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62061 req_id = 39a9b09c-2c49-4a80-98be-5f11dafc0053
60382025-04-30T06:42:35.550ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
60392025-04-30T06:42:35.551ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60402025-04-30T06:42:35.551ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62061 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 39a9b09c-2c49-4a80-98be-5f11dafc0053
60412025-04-30T06:42:35.551ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4220, tv_nsec: 350354803 } })
60422025-04-30T06:42:35.551ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
60432025-04-30T06:42:35.552ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60442025-04-30T06:42:35.552ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60452025-04-30T06:42:35.553ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fc 8c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56889 req_id = 7b2047bf-f502-4561-ba93-758b4358f186
60462025-04-30T06:42:35.553ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64652,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56889 req_id = 7b2047bf-f502-4561-ba93-758b4358f186
60472025-04-30T06:42:35.553ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60482025-04-30T06:42:35.553ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56889 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7b2047bf-f502-4561-ba93-758b4358f186
60492025-04-30T06:42:35.638ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
60502025-04-30T06:42:35.638ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60512025-04-30T06:42:35.639ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60522025-04-30T06:42:35.639ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60532025-04-30T06:42:35.639ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60542025-04-30T06:42:35.640ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60552025-04-30T06:42:35.641ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60562025-04-30T06:42:35.641ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60572025-04-30T06:42:35.641ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
60582025-04-30T06:42:35.641ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60592025-04-30T06:42:35.641ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e3 37 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .7..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58578 req_id = 60997ccb-55bc-40dc-b4c5-4f88ab623123
60602025-04-30T06:42:35.641ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58167,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58578 req_id = 60997ccb-55bc-40dc-b4c5-4f88ab623123
60612025-04-30T06:42:35.641ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
60622025-04-30T06:42:35.642ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58167, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58578 req_id = 60997ccb-55bc-40dc-b4c5-4f88ab623123
60632025-04-30T06:42:35.643ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
60642025-04-30T06:42:35.643ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
60652025-04-30T06:42:35.643ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
60662025-04-30T06:42:35.645ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60672025-04-30T06:42:35.649ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60682025-04-30T06:42:35.649ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60692025-04-30T06:42:35.649ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
60702025-04-30T06:42:35.650ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60712025-04-30T06:42:35.652ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fb463ad-c71e-48a1-9347-2b14b1576253", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60722025-04-30T06:42:35.652ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
60732025-04-30T06:42:35.654ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60742025-04-30T06:42:35.654ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
60752025-04-30T06:42:35.654ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60762025-04-30T06:42:35.664ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54480a12-f725-46a2-adf4-16a6fd9e7c90", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60772025-04-30T06:42:35.664ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
60782025-04-30T06:42:35.672ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10fc94c5-0c81-4e93-9747-5c595fb60e6f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60792025-04-30T06:42:35.672ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
60802025-04-30T06:42:35.683ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d0105b-8cff-4cc1-a4b4-ab63f3107413", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60812025-04-30T06:42:35.683ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
60822025-04-30T06:42:35.687ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45ce77a1-2853-4b11-bfa6-caa1c2307ac8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60832025-04-30T06:42:35.687ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
60842025-04-30T06:42:35.690ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2902e3be-718c-486f-89eb-9c760e53ef59", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60852025-04-30T06:42:35.690ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
60862025-04-30T06:42:35.693ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a82ad2b-7f4c-48b0-be10-60334d7642e6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60872025-04-30T06:42:35.693ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
60882025-04-30T06:42:35.713ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef449c60-5707-4d06-9bf2-022a41876c41", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60892025-04-30T06:42:35.715ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
60902025-04-30T06:42:35.715ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
60912025-04-30T06:42:35.718ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "880db632-2ec0-42df-ac73-3684771a543b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60922025-04-30T06:42:35.718ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
60932025-04-30T06:42:35.719ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b3bd5f0-2cc9-4a8c-978b-e69eecc355e8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60942025-04-30T06:42:35.719ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
60952025-04-30T06:42:35.720ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be7a7dff-f02b-4cdd-88ac-a74e3e08d816", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60962025-04-30T06:42:35.720ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
60972025-04-30T06:42:35.721ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f4d943c-9ec5-4f9f-98ae-83d98590ed8f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
60982025-04-30T06:42:35.722ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
60992025-04-30T06:42:35.723ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f672ff44-0833-4ffb-84a9-0cfa43624035", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61002025-04-30T06:42:35.723ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
61012025-04-30T06:42:35.729ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e59a92e-7ef6-4848-b4fb-0d5325e7d103", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61022025-04-30T06:42:35.729ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
61032025-04-30T06:42:35.733ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ade8431-e4eb-4edf-bba0-910c3d172358", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61042025-04-30T06:42:35.733ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
61052025-04-30T06:42:35.737ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc3d65ae-b9da-4792-913b-e4a8bfd2a57c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61062025-04-30T06:42:35.737ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
61072025-04-30T06:42:35.740ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a9ff39d-415e-484d-b1b8-ba33da79d4f0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61082025-04-30T06:42:35.740ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
61092025-04-30T06:42:35.742ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d01aa21f-9a08-4a59-86e9-f8807262baa6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61102025-04-30T06:42:35.742ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
61112025-04-30T06:42:35.744ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f92fc2a4-efc2-4dd9-afb1-a38d5852e177", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61122025-04-30T06:42:35.744ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
61132025-04-30T06:42:35.745ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5370a21-18de-4a81-b739-fa8f5eb56ecf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61142025-04-30T06:42:35.745ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
61152025-04-30T06:42:35.746ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33b21215-fd2a-4d14-9f6a-b5ced22f75f2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61162025-04-30T06:42:35.746ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
61172025-04-30T06:42:35.747ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d918a431-f42e-4610-9567-f85a4134eac5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61182025-04-30T06:42:35.748ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
61192025-04-30T06:42:35.750ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76dd6780-7818-4dbb-ac58-e3aba8a627ce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61202025-04-30T06:42:35.750ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
61212025-04-30T06:42:35.752ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "590f519e-ec4f-4eaa-9f56-0c21a7d69c78", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61222025-04-30T06:42:35.752ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
61232025-04-30T06:42:35.754ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c23be5c-8b78-4c03-b251-4bdd565d2653", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61242025-04-30T06:42:35.754ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
61252025-04-30T06:42:35.756ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "575e9f22-baf5-43af-b135-f7aa30f1d95b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61262025-04-30T06:42:35.756ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
61272025-04-30T06:42:35.758ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea14c68f-fd67-4a51-a7e1-d721eee6d950", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61282025-04-30T06:42:35.758ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
61292025-04-30T06:42:35.760ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2f90e19-9acc-489b-8186-eea164e53ca1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61302025-04-30T06:42:35.761ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
61312025-04-30T06:42:35.763ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "909251a5-eced-4df1-8e76-9b439773cf42", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61322025-04-30T06:42:35.763ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
61332025-04-30T06:42:35.765ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "307e4f28-e78b-4b73-bf28-7833c6fb1c95", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61342025-04-30T06:42:35.765ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
61352025-04-30T06:42:35.766ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af14778f-395b-4a4c-9877-fa222d3dc538", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61362025-04-30T06:42:35.766ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
61372025-04-30T06:42:35.766ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61382025-04-30T06:42:35.766ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61392025-04-30T06:42:35.766ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61402025-04-30T06:42:35.766ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
61412025-04-30T06:42:35.767ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61422025-04-30T06:42:35.768ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6838a79e-b980-481e-90e6-e1b6e2016564", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61432025-04-30T06:42:35.768ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
61442025-04-30T06:42:35.769ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c526bbf5-2d66-493a-9d9f-48383cd0234e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61452025-04-30T06:42:35.769ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
61462025-04-30T06:42:35.770ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61472025-04-30T06:42:35.774ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61482025-04-30T06:42:35.774ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61492025-04-30T06:42:35.774ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
61502025-04-30T06:42:35.775ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61512025-04-30T06:42:35.775ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "927445eb-43e2-422b-a663-048926fde263", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61522025-04-30T06:42:35.775ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
61532025-04-30T06:42:35.776ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76e02643-29e6-4038-a0ab-cf386960b0a2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61542025-04-30T06:42:35.777ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
61552025-04-30T06:42:35.777ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61562025-04-30T06:42:35.778ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
61572025-04-30T06:42:35.778ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61582025-04-30T06:42:35.779ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94402ef9-dd1e-4e17-bfb9-746516d6a5e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61592025-04-30T06:42:35.779ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
61602025-04-30T06:42:35.781ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7364833-ddf7-4b50-993a-9b713546bcb5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61612025-04-30T06:42:35.781ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
61622025-04-30T06:42:35.784ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f795ca63-231d-4743-b50b-069623312628", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61632025-04-30T06:42:35.784ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
61642025-04-30T06:42:35.786ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40e188b4-97e1-4f51-be12-b29cc6393ebd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61652025-04-30T06:42:35.786ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
61662025-04-30T06:42:35.789ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cf1e5c9-2002-4ee8-9ef2-aec2acccdc85", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61672025-04-30T06:42:35.789ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
61682025-04-30T06:42:35.792ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4b1243a-9306-4ac1-a3d4-5b755b973cfa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61692025-04-30T06:42:35.792ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
61702025-04-30T06:42:35.793ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0e497bf-bd09-494e-ad10-853f8a5a55ce", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61712025-04-30T06:42:35.793ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
61722025-04-30T06:42:35.798ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51e09a01-a2b6-4971-ae2a-5d63e952eab4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61732025-04-30T06:42:35.798ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
61742025-04-30T06:42:35.798ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52604b3d-0b45-444a-b2d9-388bb20d802e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61752025-04-30T06:42:35.798ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
61762025-04-30T06:42:35.798ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66ca63c6-6194-4291-8724-16867e8f08c8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
61772025-04-30T06:42:35.798ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61782025-04-30T06:42:35.798ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61792025-04-30T06:42:35.798ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61802025-04-30T06:42:35.798ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61812025-04-30T06:42:35.798ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61822025-04-30T06:42:35.804ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61832025-04-30T06:42:35.808ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61842025-04-30T06:42:35.809ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61852025-04-30T06:42:35.809ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61862025-04-30T06:42:35.809ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61872025-04-30T06:42:35.809ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61882025-04-30T06:42:35.810ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61892025-04-30T06:42:35.814ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61902025-04-30T06:42:35.825ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61912025-04-30T06:42:35.825ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61922025-04-30T06:42:35.825ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61932025-04-30T06:42:35.826ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61942025-04-30T06:42:35.827ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
61952025-04-30T06:42:35.884ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
61962025-04-30T06:42:35.884ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
61972025-04-30T06:42:35.884ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61982025-04-30T06:42:35.884ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61992025-04-30T06:42:35.884ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62002025-04-30T06:42:35.884ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
62012025-04-30T06:42:35.885ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62022025-04-30T06:42:35.885ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:57781 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
62032025-04-30T06:42:35.887ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:57781 req_id = bdeb5648-ea3a-4886-9cf9-6ed406d986c5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62042025-04-30T06:42:35.888ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 495 local_addr = [::1]:62607 method = GET remote_addr = [::1]:57781 req_id = bdeb5648-ea3a-4886-9cf9-6ed406d986c5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62052025-04-30T06:42:35.889ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdeb5648-ea3a-4886-9cf9-6ed406d986c5", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
62062025-04-30T06:42:35.891ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62072025-04-30T06:42:35.891ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62082025-04-30T06:42:35.891ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
62092025-04-30T06:42:35.894ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62102025-04-30T06:42:35.895ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62112025-04-30T06:42:35.895ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62122025-04-30T06:42:35.895ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62132025-04-30T06:42:35.897ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62142025-04-30T06:42:35.901ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62152025-04-30T06:42:35.909ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62162025-04-30T06:42:35.909ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62172025-04-30T06:42:35.909ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
62182025-04-30T06:42:35.909ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62192025-04-30T06:42:35.910ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62202025-04-30T06:42:35.910ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62212025-04-30T06:42:35.910ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
62222025-04-30T06:42:35.911ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
62232025-04-30T06:42:35.913ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62242025-04-30T06:42:35.916ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62252025-04-30T06:42:35.916ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62262025-04-30T06:42:35.916ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62272025-04-30T06:42:35.917ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62282025-04-30T06:42:35.918ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62292025-04-30T06:42:35.918ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
62302025-04-30T06:42:35.919ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62312025-04-30T06:42:35.921ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
62322025-04-30T06:42:35.923ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
62332025-04-30T06:42:35.930ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
62342025-04-30T06:42:35.930ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62352025-04-30T06:42:35.957ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
62362025-04-30T06:42:35.958ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
62372025-04-30T06:42:35.958ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
62382025-04-30T06:42:35.958ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 427.129718ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
62392025-04-30T06:42:35.958ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
62402025-04-30T06:42:35.958ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62412025-04-30T06:42:35.958ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62422025-04-30T06:42:35.958ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62432025-04-30T06:42:35.958ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62442025-04-30T06:42:35.960ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62452025-04-30T06:42:35.964ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62462025-04-30T06:42:35.973ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62472025-04-30T06:42:35.973ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62482025-04-30T06:42:35.973ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62492025-04-30T06:42:35.974ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62502025-04-30T06:42:35.979ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
62512025-04-30T06:42:35.979ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62522025-04-30T06:42:35.981ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 65 91 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 e............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60410 req_id = 94750740-1669-467f-8fd5-1b7247546a80
62532025-04-30T06:42:35.981ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26001,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60410 req_id = 94750740-1669-467f-8fd5-1b7247546a80
62542025-04-30T06:42:35.981ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
62552025-04-30T06:42:35.981ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62562025-04-30T06:42:35.981ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60410 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 94750740-1669-467f-8fd5-1b7247546a80
62572025-04-30T06:42:35.982ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4220, tv_nsec: 781136708 } })
62582025-04-30T06:42:35.982ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
62592025-04-30T06:42:35.982ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1b 17 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49718 req_id = 5234eb69-7585-4a40-9e64-9367eb9b7809
62602025-04-30T06:42:35.982ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6935,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49718 req_id = 5234eb69-7585-4a40-9e64-9367eb9b7809
62612025-04-30T06:42:35.982ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62622025-04-30T06:42:35.982ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49718 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5234eb69-7585-4a40-9e64-9367eb9b7809
62632025-04-30T06:42:36.030ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
62642025-04-30T06:42:36.030ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62652025-04-30T06:42:36.030ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62662025-04-30T06:42:36.030ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62672025-04-30T06:42:36.030ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
62682025-04-30T06:42:36.030ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62692025-04-30T06:42:36.031ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
62702025-04-30T06:42:36.032ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a0 7d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .}..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55756 req_id = 369dd6e7-d1ac-4440-888d-dbd1fa388a5a
62712025-04-30T06:42:36.032ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41085,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55756 req_id = 369dd6e7-d1ac-4440-888d-dbd1fa388a5a
62722025-04-30T06:42:36.032ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
62732025-04-30T06:42:36.032ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41085, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55756 req_id = 369dd6e7-d1ac-4440-888d-dbd1fa388a5a
62742025-04-30T06:42:36.035ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
62752025-04-30T06:42:36.035ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
62762025-04-30T06:42:36.035ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
62772025-04-30T06:42:36.035ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62782025-04-30T06:42:36.044ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62792025-04-30T06:42:36.044ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62802025-04-30T06:42:36.044ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
62812025-04-30T06:42:36.047ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62822025-04-30T06:42:36.051ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62832025-04-30T06:42:36.051ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
62842025-04-30T06:42:36.051ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62852025-04-30T06:42:36.055ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9c6b4d9-bad7-4ab1-abff-eaa98cbaf890", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
62862025-04-30T06:42:36.055ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
62872025-04-30T06:42:36.058ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59332701-d470-46cb-9069-2fc8c51c957c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
62882025-04-30T06:42:36.058ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
62892025-04-30T06:42:36.063ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b040176e-55ab-4c7a-9560-46c99de22213", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
62902025-04-30T06:42:36.064ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
62912025-04-30T06:42:36.132ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a7fc682-108f-4c17-8770-4ee23ee7ea52", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
62922025-04-30T06:42:36.132ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
62932025-04-30T06:42:36.156ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
62942025-04-30T06:42:36.158ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00af0b50-6032-4af0-8786-660e890b2a91", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
62952025-04-30T06:42:36.158ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
62962025-04-30T06:42:36.160ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b287ccd-b7c9-4c6b-a6c2-f1e0411027ee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
62972025-04-30T06:42:36.160ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
62982025-04-30T06:42:36.162ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13c0880a-c431-4590-b26d-a7d1d99df0da", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
62992025-04-30T06:42:36.162ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
63002025-04-30T06:42:36.166ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55cb30be-e771-4711-ba59-7c169196765f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63012025-04-30T06:42:36.167ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
63022025-04-30T06:42:36.168ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec4a20f5-35dc-46e0-b468-db2cae9c46a2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63032025-04-30T06:42:36.168ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
63042025-04-30T06:42:36.169ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0523939-9347-411f-83fe-d0a39861b070", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63052025-04-30T06:42:36.169ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
63062025-04-30T06:42:36.170ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f4d2e3e-7252-451b-a378-7cb4f9bf378e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63072025-04-30T06:42:36.170ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
63082025-04-30T06:42:36.171ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e4431e1-ec71-4e5c-89c2-9ae2bc27b590", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63092025-04-30T06:42:36.171ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
63102025-04-30T06:42:36.172ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0549a65-36ca-413a-94c4-8ce01eab0d13", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63112025-04-30T06:42:36.172ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
63122025-04-30T06:42:36.175ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "266b99be-92b2-4bb3-a7da-5c3dede775d0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63132025-04-30T06:42:36.175ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
63142025-04-30T06:42:36.176ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eca1826-8afb-4249-a33f-5ed74011999d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63152025-04-30T06:42:36.176ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
63162025-04-30T06:42:36.180ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e8545e6-5561-48a5-b51f-e6726f2622d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63172025-04-30T06:42:36.180ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
63182025-04-30T06:42:36.181ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bbbd075-4e75-4853-b730-e101f44e1192", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63192025-04-30T06:42:36.181ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
63202025-04-30T06:42:36.183ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54325e2d-59b2-4009-b5f3-af05026c0e09", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63212025-04-30T06:42:36.183ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
63222025-04-30T06:42:36.186ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08cf1689-8f61-497d-8f29-36ba77366850", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63232025-04-30T06:42:36.186ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
63242025-04-30T06:42:36.187ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5094c9c-8d83-4423-a1fd-d8e8492036a0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63252025-04-30T06:42:36.187ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
63262025-04-30T06:42:36.188ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b7d063-f2fe-4d5c-84b0-730865158157", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63272025-04-30T06:42:36.188ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
63282025-04-30T06:42:36.189ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d083a810-a40c-406f-9373-5bcf536da89a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63292025-04-30T06:42:36.189ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
63302025-04-30T06:42:36.190ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5c3ff47-3b96-40eb-aef3-c1e8ce4818c6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63312025-04-30T06:42:36.190ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
63322025-04-30T06:42:36.192ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18d86250-7919-48b7-953e-183d2d39b3e5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63332025-04-30T06:42:36.208ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
63342025-04-30T06:42:36.208ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63352025-04-30T06:42:36.208ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63362025-04-30T06:42:36.208ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63372025-04-30T06:42:36.208ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
63382025-04-30T06:42:36.209ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63392025-04-30T06:42:36.216ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63402025-04-30T06:42:36.234ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63412025-04-30T06:42:36.234ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63422025-04-30T06:42:36.234ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
63432025-04-30T06:42:36.235ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63442025-04-30T06:42:36.238ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63452025-04-30T06:42:36.238ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
63462025-04-30T06:42:36.239ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63472025-04-30T06:42:36.240ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99cc0b30-1421-46f7-b3ce-2d18cc54c065", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63482025-04-30T06:42:36.240ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
63492025-04-30T06:42:36.244ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd7e5389-c117-48e3-8f4b-400a727d2266", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63502025-04-30T06:42:36.244ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
63512025-04-30T06:42:36.256ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c3fc449-88a0-4680-85cb-2406fe3924b1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63522025-04-30T06:42:36.256ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
63532025-04-30T06:42:36.259ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bada40d-19ca-4a00-a173-3640efbe821c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63542025-04-30T06:42:36.259ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
63552025-04-30T06:42:36.265ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c55232f9-ad67-4435-bfd3-53251ed4a88e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63562025-04-30T06:42:36.265ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
63572025-04-30T06:42:36.266ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd0cd25b-92ae-40da-bd10-20f765d917ad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63582025-04-30T06:42:36.266ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
63592025-04-30T06:42:36.268ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fe700f6-a9eb-481c-bc78-55e200fbfde7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63602025-04-30T06:42:36.268ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
63612025-04-30T06:42:36.274ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef1bd9d6-7606-474e-b5f8-576e1b939cb6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63622025-04-30T06:42:36.274ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
63632025-04-30T06:42:36.276ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07ab6a2e-de5f-4017-976f-3ebeaddfebf4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63642025-04-30T06:42:36.276ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
63652025-04-30T06:42:36.277ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "975c6a58-04b6-4f07-b986-fea6b84c1bdf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63662025-04-30T06:42:36.277ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
63672025-04-30T06:42:36.279ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a2b2dd-6d2a-4ba7-bb18-e014eec4f957", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63682025-04-30T06:42:36.280ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
63692025-04-30T06:42:36.300ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
63702025-04-30T06:42:36.303ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a0b082e-4bcf-436f-912e-d28977e954e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63712025-04-30T06:42:36.303ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
63722025-04-30T06:42:36.306ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02838296-d06c-4d99-81e7-56f1e98baca6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63732025-04-30T06:42:36.307ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
63742025-04-30T06:42:36.309ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "874b2d84-07b6-4e63-872f-c313df763931", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63752025-04-30T06:42:36.309ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
63762025-04-30T06:42:36.311ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33e96b84-2336-4f69-b265-dc811d5ffb51", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63772025-04-30T06:42:36.311ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
63782025-04-30T06:42:36.312ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39485760-986a-4b0d-b9d4-d869c2b9e743", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63792025-04-30T06:42:36.313ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
63802025-04-30T06:42:36.329ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce5aa41f-4ee8-44fa-ade1-12fc43c53c29", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63812025-04-30T06:42:36.330ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
63822025-04-30T06:42:36.330ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d671d930-ff71-4898-ba44-a841415f7971", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63832025-04-30T06:42:36.330ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
63842025-04-30T06:42:36.331ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a410c6b8-855c-4bc0-a30c-49c0b6e94687", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63852025-04-30T06:42:36.331ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
63862025-04-30T06:42:36.331ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a15aebe0-6b2b-4da8-bf76-5c2a7d9aae22", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63872025-04-30T06:42:36.331ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
63882025-04-30T06:42:36.331ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd3f9a01-98f2-435d-886b-906ce751856e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
63892025-04-30T06:42:36.340ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63902025-04-30T06:42:36.340ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63912025-04-30T06:42:36.341ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63922025-04-30T06:42:36.341ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63932025-04-30T06:42:36.341ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63942025-04-30T06:42:36.341ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63952025-04-30T06:42:36.341ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63962025-04-30T06:42:36.341ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63972025-04-30T06:42:36.341ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63982025-04-30T06:42:36.341ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63992025-04-30T06:42:36.341ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64002025-04-30T06:42:36.341ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64012025-04-30T06:42:36.342ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64022025-04-30T06:42:36.348ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
64032025-04-30T06:42:36.348ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64042025-04-30T06:42:36.348ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64052025-04-30T06:42:36.351ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64062025-04-30T06:42:36.355ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64072025-04-30T06:42:36.355ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64082025-04-30T06:42:36.355ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64092025-04-30T06:42:36.355ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
64102025-04-30T06:42:36.355ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64112025-04-30T06:42:36.364ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64122025-04-30T06:42:36.372ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64132025-04-30T06:42:36.373ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64142025-04-30T06:42:36.373ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
64152025-04-30T06:42:36.374ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64162025-04-30T06:42:36.472ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
64172025-04-30T06:42:36.472ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
64182025-04-30T06:42:36.473ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:40529 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
64192025-04-30T06:42:36.477ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64202025-04-30T06:42:36.487ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
64212025-04-30T06:42:36.488ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64222025-04-30T06:42:36.489ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:40529 req_id = fa0dae25-0b69-499b-b178-fc0f01b75421 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64232025-04-30T06:42:36.509ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 19758 local_addr = [::1]:62607 method = GET remote_addr = [::1]:40529 req_id = fa0dae25-0b69-499b-b178-fc0f01b75421 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64242025-04-30T06:42:36.510ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa0dae25-0b69-499b-b178-fc0f01b75421", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:35 GMT"} })
64252025-04-30T06:42:36.545ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64262025-04-30T06:42:36.545ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64272025-04-30T06:42:36.545ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
64282025-04-30T06:42:36.547ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64292025-04-30T06:42:36.547ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64302025-04-30T06:42:36.547ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64312025-04-30T06:42:36.547ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64322025-04-30T06:42:36.548ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64332025-04-30T06:42:36.553ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64342025-04-30T06:42:36.559ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64352025-04-30T06:42:36.559ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64362025-04-30T06:42:36.559ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64372025-04-30T06:42:36.561ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64382025-04-30T06:42:36.570ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64392025-04-30T06:42:36.570ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64402025-04-30T06:42:36.572ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64412025-04-30T06:42:36.572ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64422025-04-30T06:42:36.573ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64432025-04-30T06:42:36.573ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64442025-04-30T06:42:36.616ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
64452025-04-30T06:42:36.621ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
64462025-04-30T06:42:36.622ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
64472025-04-30T06:42:36.622ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 663.574739ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
64482025-04-30T06:42:36.622ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
64492025-04-30T06:42:36.622ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64502025-04-30T06:42:36.622ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64512025-04-30T06:42:36.622ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64522025-04-30T06:42:36.622ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64532025-04-30T06:42:36.623ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64542025-04-30T06:42:36.625ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64552025-04-30T06:42:36.628ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64562025-04-30T06:42:36.628ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64572025-04-30T06:42:36.628ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64582025-04-30T06:42:36.628ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64592025-04-30T06:42:36.631ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
64602025-04-30T06:42:36.631ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
64612025-04-30T06:42:36.633ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 33 6a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 3j..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49910 req_id = 0d4ee4f7-92fa-48aa-a196-9f04dbc06e9b
64622025-04-30T06:42:36.633ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13162,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49910 req_id = 0d4ee4f7-92fa-48aa-a196-9f04dbc06e9b
64632025-04-30T06:42:36.633ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
64642025-04-30T06:42:36.633ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64652025-04-30T06:42:36.633ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49910 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0d4ee4f7-92fa-48aa-a196-9f04dbc06e9b
64662025-04-30T06:42:36.633ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4221, tv_nsec: 432326772 } })
64672025-04-30T06:42:36.633ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
64682025-04-30T06:42:36.633ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 55 3b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e U;..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63781 req_id = 2d181e8e-e94c-45ef-9870-13f63417e25c
64692025-04-30T06:42:36.633ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21819,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63781 req_id = 2d181e8e-e94c-45ef-9870-13f63417e25c
64702025-04-30T06:42:36.633ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64712025-04-30T06:42:36.634ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63781 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2d181e8e-e94c-45ef-9870-13f63417e25c
64722025-04-30T06:42:36.768ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
64732025-04-30T06:42:36.769ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64742025-04-30T06:42:36.769ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64752025-04-30T06:42:36.769ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64762025-04-30T06:42:36.769ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
64772025-04-30T06:42:36.770ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64782025-04-30T06:42:36.771ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 36 67 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 6g..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64083 req_id = 689c099e-3bd1-4b29-8fdd-884e75768f9b
64792025-04-30T06:42:36.771ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13927,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64083 req_id = 689c099e-3bd1-4b29-8fdd-884e75768f9b
64802025-04-30T06:42:36.771ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
64812025-04-30T06:42:36.771ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13927, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64083 req_id = 689c099e-3bd1-4b29-8fdd-884e75768f9b
64822025-04-30T06:42:36.772ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
64832025-04-30T06:42:36.772ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
64842025-04-30T06:42:36.772ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
64852025-04-30T06:42:36.777ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64862025-04-30T06:42:36.790ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64872025-04-30T06:42:36.790ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64882025-04-30T06:42:36.790ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
64892025-04-30T06:42:36.791ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64902025-04-30T06:42:36.795ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64912025-04-30T06:42:36.795ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
64922025-04-30T06:42:36.796ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64932025-04-30T06:42:36.797ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddb78a3d-e369-4bc8-831f-db373cfc9e58", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
64942025-04-30T06:42:36.798ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
64952025-04-30T06:42:36.800ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f096f4e3-aa9e-4b2f-bd39-f1e65995a7ee", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
64962025-04-30T06:42:36.801ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
64972025-04-30T06:42:36.806ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "508923fc-bff4-493b-bc0e-66cb385f1411", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
64982025-04-30T06:42:36.807ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
64992025-04-30T06:42:36.811ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b726d06b-0324-430b-bf79-c93bc6648007", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65002025-04-30T06:42:36.813ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
65012025-04-30T06:42:36.820ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b74a263b-72f2-4322-b8c8-997c62c63e07", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65022025-04-30T06:42:36.820ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
65032025-04-30T06:42:36.823ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "872bc5be-c2ab-43aa-a790-5f08a8fa8ef2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65042025-04-30T06:42:36.823ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
65052025-04-30T06:42:36.827ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93611bf2-6837-41ce-b26b-d33d6bc7d102", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65062025-04-30T06:42:36.827ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
65072025-04-30T06:42:36.830ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a58ecfab-8cdf-4ea6-9511-3eecb407119c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65082025-04-30T06:42:36.830ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
65092025-04-30T06:42:36.838ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
65102025-04-30T06:42:36.838ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7c07af7-bd31-4f64-8fe3-7f3a886af053", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65112025-04-30T06:42:36.838ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
65122025-04-30T06:42:36.842ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "102f8ca7-ebef-49e4-8e4e-9430103d1781", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65132025-04-30T06:42:36.842ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
65142025-04-30T06:42:36.845ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2964c75f-1903-4200-ae77-f95ea27adcf0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65152025-04-30T06:42:36.846ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
65162025-04-30T06:42:36.857ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64181c29-c51e-475f-90b6-cd0d1cbbc32d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65172025-04-30T06:42:36.857ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
65182025-04-30T06:42:36.860ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e73172fd-97c5-4251-b60f-c128fa006afe", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65192025-04-30T06:42:36.867ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
65202025-04-30T06:42:36.874ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0165eb7b-586c-4078-b3c7-36ea65a5fed1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65212025-04-30T06:42:36.874ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
65222025-04-30T06:42:36.876ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63b6bb36-a120-45e5-b1c7-8c7f8d35616a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65232025-04-30T06:42:36.876ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
65242025-04-30T06:42:36.881ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a233a84e-918c-42b7-8a79-fc699c1c61fd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65252025-04-30T06:42:36.881ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
65262025-04-30T06:42:36.883ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7765082b-0294-4c88-aa6f-3b88da07a51f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65272025-04-30T06:42:36.883ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
65282025-04-30T06:42:36.885ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "651a0a6f-2b74-4f0c-a006-89996a5d344b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65292025-04-30T06:42:36.885ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
65302025-04-30T06:42:36.886ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8cdaa3a-4663-4bd9-a0e6-0c9c47391948", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65312025-04-30T06:42:36.887ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
65322025-04-30T06:42:36.888ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8e9123c-8686-4d83-9d4a-8b28d2b19cfc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65332025-04-30T06:42:36.888ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
65342025-04-30T06:42:36.889ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65352025-04-30T06:42:36.889ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65362025-04-30T06:42:36.889ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65372025-04-30T06:42:36.889ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
65382025-04-30T06:42:36.890ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65392025-04-30T06:42:36.891ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8907e7f-c6cd-4eb7-a751-9560efa489d5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65402025-04-30T06:42:36.891ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
65412025-04-30T06:42:36.892ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36fb1fc2-a8b8-40fb-9b75-87b137a4c2f9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65422025-04-30T06:42:36.892ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
65432025-04-30T06:42:36.893ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65442025-04-30T06:42:36.903ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65452025-04-30T06:42:36.903ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65462025-04-30T06:42:36.903ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
65472025-04-30T06:42:36.903ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65482025-04-30T06:42:36.904ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
65492025-04-30T06:42:36.904ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
65502025-04-30T06:42:36.905ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
65512025-04-30T06:42:36.906ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
65522025-04-30T06:42:36.907ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1d5fde5-2dea-4aca-95b5-87ae93a77252", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65532025-04-30T06:42:36.908ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
65542025-04-30T06:42:36.908ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65552025-04-30T06:42:36.908ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
65562025-04-30T06:42:36.908ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65572025-04-30T06:42:36.911ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a5f7ae5-c7fb-47eb-82e2-1f9a7ba77eec", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65582025-04-30T06:42:36.911ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
65592025-04-30T06:42:36.911ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
65602025-04-30T06:42:36.911ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
65612025-04-30T06:42:36.915ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cd2e9e0-0c89-4f05-b3eb-cb32395f695a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65622025-04-30T06:42:36.915ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
65632025-04-30T06:42:36.915ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
65642025-04-30T06:42:36.915ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
65652025-04-30T06:42:36.918ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b8d0a80-c934-4b72-a661-4e5e0b40fda4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65662025-04-30T06:42:36.918ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
65672025-04-30T06:42:36.922ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be7bc7ce-c6f6-4338-93af-316d1738c7ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65682025-04-30T06:42:36.922ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
65692025-04-30T06:42:36.927ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1d8717c-e5cf-4851-964c-1c8b1aabad8b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65702025-04-30T06:42:36.927ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
65712025-04-30T06:42:36.929ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb426789-c57d-4467-b349-18fa100f9f75", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65722025-04-30T06:42:36.929ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
65732025-04-30T06:42:36.933ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c39b0e24-ce99-40e0-899d-c19712d4f9df", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65742025-04-30T06:42:36.933ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
65752025-04-30T06:42:36.935ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7c25c09-1845-4b68-a29a-192fe471b89a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65762025-04-30T06:42:36.935ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
65772025-04-30T06:42:36.937ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78face54-07e6-4327-a395-acb14313a7f1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65782025-04-30T06:42:36.937ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
65792025-04-30T06:42:36.939ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9580f557-8c52-4a43-a32d-d7ffacf666a6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65802025-04-30T06:42:36.940ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
65812025-04-30T06:42:36.951ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
65822025-04-30T06:42:36.951ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6559a88-9a59-486e-b088-749f2aae060c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65832025-04-30T06:42:36.951ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
65842025-04-30T06:42:36.953ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38682e08-7d74-4713-8f94-a4f8565c3b0d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65852025-04-30T06:42:36.955ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
65862025-04-30T06:42:36.957ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75f92101-2856-4650-984b-ad50426391ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65872025-04-30T06:42:36.957ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
65882025-04-30T06:42:36.960ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d99d5c0e-8d9e-44d7-9520-ec9db502cf3e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65892025-04-30T06:42:36.960ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
65902025-04-30T06:42:36.962ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e19f9ff-cbfb-4510-973d-b7fb884f9be3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65912025-04-30T06:42:36.962ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
65922025-04-30T06:42:36.964ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec7e2f6-cc18-472c-8910-fe64c53e4262", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65932025-04-30T06:42:36.964ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
65942025-04-30T06:42:36.967ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11cad9b2-b34b-4ce0-bff8-60f7e6093ba1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65952025-04-30T06:42:36.967ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
65962025-04-30T06:42:36.976ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b79372e6-4d9d-4241-9d65-ce4b0a9f57f3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65972025-04-30T06:42:36.977ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
65982025-04-30T06:42:36.978ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f32df93-a075-4f89-ba80-06a4c6fc23fc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
65992025-04-30T06:42:36.978ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
66002025-04-30T06:42:36.979ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e644f7b9-0fbe-48ea-be7b-ba7916020cc8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
66012025-04-30T06:42:36.979ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
66022025-04-30T06:42:36.980ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74cd3c29-c6ce-46fa-9fe0-dec38f085a9a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
66032025-04-30T06:42:36.980ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
66042025-04-30T06:42:36.981ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d7e09eb-ef61-491f-ace4-3493571b2579", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
66052025-04-30T06:42:36.981ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66062025-04-30T06:42:36.981ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66072025-04-30T06:42:36.981ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66082025-04-30T06:42:36.981ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66092025-04-30T06:42:36.982ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66102025-04-30T06:42:37.021ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66112025-04-30T06:42:37.021ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66122025-04-30T06:42:37.021ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66132025-04-30T06:42:37.021ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
66142025-04-30T06:42:37.022ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66152025-04-30T06:42:37.022ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
66162025-04-30T06:42:37.033ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66172025-04-30T06:42:37.036ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
66182025-04-30T06:42:37.036ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66192025-04-30T06:42:37.036ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66202025-04-30T06:42:37.036ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66212025-04-30T06:42:37.036ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66222025-04-30T06:42:37.037ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66232025-04-30T06:42:37.039ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66242025-04-30T06:42:37.047ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66252025-04-30T06:42:37.047ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66262025-04-30T06:42:37.047ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
66272025-04-30T06:42:37.048ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66282025-04-30T06:42:37.048ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66292025-04-30T06:42:37.051ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
66302025-04-30T06:42:37.051ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66312025-04-30T06:42:37.051ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66322025-04-30T06:42:37.052ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66332025-04-30T06:42:37.065ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66342025-04-30T06:42:37.066ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
66352025-04-30T06:42:37.067ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66362025-04-30T06:42:37.137ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
66372025-04-30T06:42:37.137ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
66382025-04-30T06:42:37.138ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:43481 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
66392025-04-30T06:42:37.142ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:43481 req_id = 25e46039-318c-4086-8496-4a3a042ed413 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66402025-04-30T06:42:37.142ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1108 local_addr = [::1]:62607 method = GET remote_addr = [::1]:43481 req_id = 25e46039-318c-4086-8496-4a3a042ed413 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66412025-04-30T06:42:37.144ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "25e46039-318c-4086-8496-4a3a042ed413", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
66422025-04-30T06:42:37.145ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66432025-04-30T06:42:37.145ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66442025-04-30T06:42:37.145ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
66452025-04-30T06:42:37.147ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66462025-04-30T06:42:37.147ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66472025-04-30T06:42:37.147ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66482025-04-30T06:42:37.147ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66492025-04-30T06:42:37.148ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66502025-04-30T06:42:37.157ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66512025-04-30T06:42:37.162ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66522025-04-30T06:42:37.163ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66532025-04-30T06:42:37.163ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66542025-04-30T06:42:37.163ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66552025-04-30T06:42:37.217ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
66562025-04-30T06:42:37.238ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
66572025-04-30T06:42:37.239ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
66582025-04-30T06:42:37.239ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 617.076414ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
66592025-04-30T06:42:37.239ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
66602025-04-30T06:42:37.239ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66612025-04-30T06:42:37.239ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66622025-04-30T06:42:37.239ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66632025-04-30T06:42:37.239ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66642025-04-30T06:42:37.240ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66652025-04-30T06:42:37.276ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66662025-04-30T06:42:37.276ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66672025-04-30T06:42:37.276ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66682025-04-30T06:42:37.276ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
66692025-04-30T06:42:37.277ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66702025-04-30T06:42:37.278ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66712025-04-30T06:42:37.286ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66722025-04-30T06:42:37.286ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66732025-04-30T06:42:37.286ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66742025-04-30T06:42:37.286ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66752025-04-30T06:42:37.292ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66762025-04-30T06:42:37.299ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66772025-04-30T06:42:37.300ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66782025-04-30T06:42:37.300ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
66792025-04-30T06:42:37.300ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66802025-04-30T06:42:37.304ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
66812025-04-30T06:42:37.307ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
66822025-04-30T06:42:37.311ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b5 2a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .*..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58312 req_id = cf02fc4c-f83f-4413-a7a3-fee7fee8fecc
66832025-04-30T06:42:37.311ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46378,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58312 req_id = cf02fc4c-f83f-4413-a7a3-fee7fee8fecc
66842025-04-30T06:42:37.311ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
66852025-04-30T06:42:37.311ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66862025-04-30T06:42:37.311ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58312 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cf02fc4c-f83f-4413-a7a3-fee7fee8fecc
66872025-04-30T06:42:37.312ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66882025-04-30T06:42:37.312ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
66892025-04-30T06:42:37.313ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66902025-04-30T06:42:37.313ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4222, tv_nsec: 112763978 } })
66912025-04-30T06:42:37.313ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
66922025-04-30T06:42:37.314ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: aa c3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60750 req_id = c8cb894d-fc84-492a-a75a-7f4940c29002
66932025-04-30T06:42:37.315ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43715,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60750 req_id = c8cb894d-fc84-492a-a75a-7f4940c29002
66942025-04-30T06:42:37.315ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66952025-04-30T06:42:37.315ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60750 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c8cb894d-fc84-492a-a75a-7f4940c29002
66962025-04-30T06:42:37.365ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
66972025-04-30T06:42:37.367ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 43 dc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c C............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64472 req_id = 2f439fec-075f-436a-9180-89a4b83f2507
66982025-04-30T06:42:37.367ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17372,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64472 req_id = 2f439fec-075f-436a-9180-89a4b83f2507
66992025-04-30T06:42:37.368ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
67002025-04-30T06:42:37.368ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17372, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64472 req_id = 2f439fec-075f-436a-9180-89a4b83f2507
67012025-04-30T06:42:37.369ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
67022025-04-30T06:42:37.369ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
67032025-04-30T06:42:37.369ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
67042025-04-30T06:42:37.376ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f2714aa-eb3b-4700-acca-c63d37edeb3f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67052025-04-30T06:42:37.376ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
67062025-04-30T06:42:37.391ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f962ab5-c6e5-4258-a2de-46d6182d9561", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67072025-04-30T06:42:37.391ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
67082025-04-30T06:42:37.399ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61a42fab-e0fa-4a16-989f-064c71a7a055", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67092025-04-30T06:42:37.399ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
67102025-04-30T06:42:37.407ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b74a1085-ccec-49f1-9771-d5599a028dc4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67112025-04-30T06:42:37.407ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
67122025-04-30T06:42:37.413ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d0a5ef6-3185-4db7-b298-856e33716c47", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67132025-04-30T06:42:37.413ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
67142025-04-30T06:42:37.421ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
67152025-04-30T06:42:37.449ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed80976b-8d92-4315-a2b6-e6d79059f926", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67162025-04-30T06:42:37.449ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
67172025-04-30T06:42:37.452ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efb34c76-b4bf-4fc7-b611-0e968f0f4174", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67182025-04-30T06:42:37.452ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
67192025-04-30T06:42:37.455ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2edb876c-358e-4bda-8cc9-0631332b49fe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67202025-04-30T06:42:37.456ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
67212025-04-30T06:42:37.460ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47758f00-4c87-48e2-a175-cfdfbe558fc9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67222025-04-30T06:42:37.464ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
67232025-04-30T06:42:37.466ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73dec4e6-2b14-423b-99e4-10dc88d6c1bf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67242025-04-30T06:42:37.466ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
67252025-04-30T06:42:37.476ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67262025-04-30T06:42:37.476ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67272025-04-30T06:42:37.476ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67282025-04-30T06:42:37.476ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
67292025-04-30T06:42:37.477ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67302025-04-30T06:42:37.478ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39f79f3b-f05a-42ee-8e63-29cbc5d8b67d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67312025-04-30T06:42:37.478ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
67322025-04-30T06:42:37.480ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cad8acff-6547-4c6f-b45f-3a514914b463", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67332025-04-30T06:42:37.480ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
67342025-04-30T06:42:37.481ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67352025-04-30T06:42:37.491ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67362025-04-30T06:42:37.491ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67372025-04-30T06:42:37.491ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
67382025-04-30T06:42:37.491ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67392025-04-30T06:42:37.494ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
67402025-04-30T06:42:37.501ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67412025-04-30T06:42:37.501ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67422025-04-30T06:42:37.501ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
67432025-04-30T06:42:37.502ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67442025-04-30T06:42:37.502ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 4abed521-74bf-45c0-85a1-558561214402 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67452025-04-30T06:42:37.502ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 4abed521-74bf-45c0-85a1-558561214402 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67462025-04-30T06:42:37.502ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 4abed521-74bf-45c0-85a1-558561214402 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67472025-04-30T06:42:37.503ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 4abed521-74bf-45c0-85a1-558561214402 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67482025-04-30T06:42:37.504ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "900227c5-d1e4-4c78-8438-b0edec1cba0e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67492025-04-30T06:42:37.504ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
67502025-04-30T06:42:37.543ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 41512 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 4abed521-74bf-45c0-85a1-558561214402 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67512025-04-30T06:42:37.544ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "4abed521-74bf-45c0-85a1-558561214402", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67522025-04-30T06:42:37.545ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
67532025-04-30T06:42:37.547ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 91c613fa-379a-4b9d-99f8-9d3f552b626c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
67542025-04-30T06:42:37.548ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 91c613fa-379a-4b9d-99f8-9d3f552b626c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
67552025-04-30T06:42:37.548ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 91c613fa-379a-4b9d-99f8-9d3f552b626c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
67562025-04-30T06:42:37.549ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 91c613fa-379a-4b9d-99f8-9d3f552b626c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
67572025-04-30T06:42:37.552ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08944369-0fec-4184-98e0-c48ff9fb401d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67582025-04-30T06:42:37.552ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
67592025-04-30T06:42:37.552ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67602025-04-30T06:42:37.552ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67612025-04-30T06:42:37.554ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 6858 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 91c613fa-379a-4b9d-99f8-9d3f552b626c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
67622025-04-30T06:42:37.556ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "91c613fa-379a-4b9d-99f8-9d3f552b626c", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67632025-04-30T06:42:37.556ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67642025-04-30T06:42:37.556ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
67652025-04-30T06:42:37.556ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
67662025-04-30T06:42:37.556ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67672025-04-30T06:42:37.557ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67682025-04-30T06:42:37.558ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67692025-04-30T06:42:37.558ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67702025-04-30T06:42:37.561ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eacd96c7-e4c0-4690-b57e-f1784d8618bd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67712025-04-30T06:42:37.561ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
67722025-04-30T06:42:37.563ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89c3d4aa-8cab-4116-8d83-09fcf526824e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67732025-04-30T06:42:37.563ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
67742025-04-30T06:42:37.571ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eaeb4b2-1343-4d38-b37f-92104f0eb9a1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67752025-04-30T06:42:37.571ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
67762025-04-30T06:42:37.580ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "add3b752-0e13-432a-aac8-6223efc5d855", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67772025-04-30T06:42:37.580ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
67782025-04-30T06:42:37.580ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
67792025-04-30T06:42:37.585ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21f2db7f-c102-4747-8841-8c8694482cf1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67802025-04-30T06:42:37.586ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
67812025-04-30T06:42:37.588ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83cef0e2-52c1-42ef-bd9f-ae8433ce50f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67822025-04-30T06:42:37.588ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
67832025-04-30T06:42:37.592ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a37da0d8-e6fb-438e-9ad0-d622200a965b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67842025-04-30T06:42:37.592ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
67852025-04-30T06:42:37.593ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c60b475e-3166-4e1d-a9ec-f374af981c15", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67862025-04-30T06:42:37.594ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
67872025-04-30T06:42:37.595ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4977bcf7-a4f0-45be-b9bd-52d5299bb425", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67882025-04-30T06:42:37.595ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
67892025-04-30T06:42:37.598ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86492f54-8bea-4350-aec6-7d79b0c70c02", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67902025-04-30T06:42:37.598ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
67912025-04-30T06:42:37.603ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe4ac83d-2b36-4cee-9042-9c6b3fd97f2c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67922025-04-30T06:42:37.603ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
67932025-04-30T06:42:37.614ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e73dceb-c37a-4f2d-beb4-05f89c8b6ee6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67942025-04-30T06:42:37.615ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
67952025-04-30T06:42:37.620ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5594aac7-690a-46e6-b4f1-e465af60f1b9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67962025-04-30T06:42:37.620ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
67972025-04-30T06:42:37.624ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4b4e354-5173-47d0-b497-b4821a14f8b9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
67982025-04-30T06:42:37.624ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
67992025-04-30T06:42:37.628ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3a71dac-0d2c-456d-861b-5d9788a38913", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68002025-04-30T06:42:37.628ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
68012025-04-30T06:42:37.631ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68022025-04-30T06:42:37.631ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68032025-04-30T06:42:37.631ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68042025-04-30T06:42:37.631ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
68052025-04-30T06:42:37.632ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68062025-04-30T06:42:37.634ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7c88fce-c4ba-4d8b-ba72-7eb6a549f187", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68072025-04-30T06:42:37.635ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
68082025-04-30T06:42:37.635ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68092025-04-30T06:42:37.643ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68102025-04-30T06:42:37.644ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68112025-04-30T06:42:37.644ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
68122025-04-30T06:42:37.644ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68132025-04-30T06:42:37.646ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e376a7e6-24ff-45cf-9520-91bd62e34f89", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68142025-04-30T06:42:37.646ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
68152025-04-30T06:42:37.647ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68162025-04-30T06:42:37.647ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
68172025-04-30T06:42:37.647ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68182025-04-30T06:42:37.648ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8895bab0-9cf2-4c85-baf4-813c282561c2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68192025-04-30T06:42:37.648ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
68202025-04-30T06:42:37.651ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fafb7dbe-1850-469a-b310-4eceb0bd7742", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68212025-04-30T06:42:37.651ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
68222025-04-30T06:42:37.652ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93e3642f-8733-465e-b63b-96bc28dd2704", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68232025-04-30T06:42:37.652ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
68242025-04-30T06:42:37.654ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee00e8d4-3d0a-4d1f-bfce-6617db45645d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68252025-04-30T06:42:37.654ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
68262025-04-30T06:42:37.658ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6012079b-a6c9-48ca-9060-ed6bd81167d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68272025-04-30T06:42:37.658ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
68282025-04-30T06:42:37.663ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64a41fed-5511-4802-bbba-9b13ad2c946e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68292025-04-30T06:42:37.663ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
68302025-04-30T06:42:37.665ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87c29935-8715-4f08-9c4d-35b37c3a6ed5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68312025-04-30T06:42:37.665ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
68322025-04-30T06:42:37.670ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aed8bd2-6b0b-442d-8288-7b5df653acae", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68332025-04-30T06:42:37.670ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
68342025-04-30T06:42:37.676ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
68352025-04-30T06:42:37.676ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad53d67c-05bb-4b82-b272-f829ba43e31e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68362025-04-30T06:42:37.676ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
68372025-04-30T06:42:37.679ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24d87d9a-976d-4c3b-8078-d64f672166cf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68382025-04-30T06:42:37.679ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
68392025-04-30T06:42:37.680ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7a249c0-e26b-4fed-8cb5-b79ef107a7de", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68402025-04-30T06:42:37.680ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
68412025-04-30T06:42:37.681ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16d0b347-ab3b-4ea0-b655-9be29c608df8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68422025-04-30T06:42:37.681ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
68432025-04-30T06:42:37.683ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f64664a5-0a73-4abe-8da9-03a81fbbf39d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68442025-04-30T06:42:37.683ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
68452025-04-30T06:42:37.685ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14b8d261-d29d-4034-a8a6-e0e6b7af46ea", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:36 GMT"} })
68462025-04-30T06:42:37.685ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68472025-04-30T06:42:37.685ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68482025-04-30T06:42:37.685ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68492025-04-30T06:42:37.685ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68502025-04-30T06:42:37.687ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68512025-04-30T06:42:37.697ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68522025-04-30T06:42:37.708ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68532025-04-30T06:42:37.708ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68542025-04-30T06:42:37.708ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68552025-04-30T06:42:37.708ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68562025-04-30T06:42:37.708ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68572025-04-30T06:42:37.708ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68582025-04-30T06:42:37.726ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68592025-04-30T06:42:37.736ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68602025-04-30T06:42:37.736ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68612025-04-30T06:42:37.736ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68622025-04-30T06:42:37.737ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68632025-04-30T06:42:37.738ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68642025-04-30T06:42:37.738ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68652025-04-30T06:42:37.738ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68662025-04-30T06:42:37.738ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
68672025-04-30T06:42:37.739ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68682025-04-30T06:42:37.807ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
68692025-04-30T06:42:37.807ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
68702025-04-30T06:42:37.807ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:51361 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
68712025-04-30T06:42:37.808ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:51361 req_id = 9591d27e-43a2-49de-a6ef-bf1eb9b49172 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68722025-04-30T06:42:37.808ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 468 local_addr = [::1]:62607 method = GET remote_addr = [::1]:51361 req_id = 9591d27e-43a2-49de-a6ef-bf1eb9b49172 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68732025-04-30T06:42:37.809ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9591d27e-43a2-49de-a6ef-bf1eb9b49172", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
68742025-04-30T06:42:37.811ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68752025-04-30T06:42:37.811ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68762025-04-30T06:42:37.811ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
68772025-04-30T06:42:37.814ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68782025-04-30T06:42:37.815ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68792025-04-30T06:42:37.815ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68802025-04-30T06:42:37.815ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68812025-04-30T06:42:37.816ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68822025-04-30T06:42:37.818ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68832025-04-30T06:42:37.824ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68842025-04-30T06:42:37.825ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68852025-04-30T06:42:37.825ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
68862025-04-30T06:42:37.826ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68872025-04-30T06:42:37.829ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68882025-04-30T06:42:37.842ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68892025-04-30T06:42:37.842ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68902025-04-30T06:42:37.843ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68912025-04-30T06:42:37.843ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68922025-04-30T06:42:37.846ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68932025-04-30T06:42:37.847ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
68942025-04-30T06:42:37.847ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68952025-04-30T06:42:37.888ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
68962025-04-30T06:42:37.888ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
68972025-04-30T06:42:37.888ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 649.003737ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
68982025-04-30T06:42:37.888ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
68992025-04-30T06:42:37.888ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
69002025-04-30T06:42:37.888ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69012025-04-30T06:42:37.888ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69022025-04-30T06:42:37.888ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69032025-04-30T06:42:37.892ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69042025-04-30T06:42:37.896ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
69052025-04-30T06:42:37.898ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69062025-04-30T06:42:37.903ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
69072025-04-30T06:42:37.903ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69082025-04-30T06:42:37.903ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69092025-04-30T06:42:37.904ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69102025-04-30T06:42:37.905ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69112025-04-30T06:42:37.905ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69122025-04-30T06:42:37.905ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
69132025-04-30T06:42:37.906ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
69142025-04-30T06:42:37.919ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
69152025-04-30T06:42:37.919ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
69162025-04-30T06:42:37.925ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 57 bf 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 W............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52222 req_id = 61c613ec-a369-4d6a-8e5f-35e609867521
69172025-04-30T06:42:37.926ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22463,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52222 req_id = 61c613ec-a369-4d6a-8e5f-35e609867521
69182025-04-30T06:42:37.926ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
69192025-04-30T06:42:37.926ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
69202025-04-30T06:42:37.926ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52222 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 61c613ec-a369-4d6a-8e5f-35e609867521
69212025-04-30T06:42:37.928ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
69222025-04-30T06:42:37.928ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
69232025-04-30T06:42:37.929ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4222, tv_nsec: 728812725 } })
69242025-04-30T06:42:37.930ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
69252025-04-30T06:42:37.934ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 19 02 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51827 req_id = 5848307e-e416-4df1-8227-563295225caf
69262025-04-30T06:42:37.934ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6402,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51827 req_id = 5848307e-e416-4df1-8227-563295225caf
69272025-04-30T06:42:37.934ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
69282025-04-30T06:42:37.934ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51827 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5848307e-e416-4df1-8227-563295225caf
69292025-04-30T06:42:38.034ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
69302025-04-30T06:42:38.034ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69312025-04-30T06:42:38.035ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69322025-04-30T06:42:38.035ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69332025-04-30T06:42:38.036ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
69342025-04-30T06:42:38.037ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69352025-04-30T06:42:38.038ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
69362025-04-30T06:42:38.038ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69372025-04-30T06:42:38.038ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
69382025-04-30T06:42:38.039ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0f d6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58366 req_id = 8e93a4a1-4200-4176-8d06-482ba82a561a
69392025-04-30T06:42:38.039ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4054,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58366 req_id = 8e93a4a1-4200-4176-8d06-482ba82a561a
69402025-04-30T06:42:38.041ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
69412025-04-30T06:42:38.041ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4054, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58366 req_id = 8e93a4a1-4200-4176-8d06-482ba82a561a
69422025-04-30T06:42:38.042ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
69432025-04-30T06:42:38.042ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
69442025-04-30T06:42:38.042ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
69452025-04-30T06:42:38.043ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69462025-04-30T06:42:38.060ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69472025-04-30T06:42:38.060ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69482025-04-30T06:42:38.060ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
69492025-04-30T06:42:38.088ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69502025-04-30T06:42:38.102ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e79900cf-ba82-4567-80b0-a755d10542f4", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69512025-04-30T06:42:38.103ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
69522025-04-30T06:42:38.103ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69532025-04-30T06:42:38.103ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
69542025-04-30T06:42:38.104ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69552025-04-30T06:42:38.139ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "133a814c-5310-4e1f-867f-bda42e9783da", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69562025-04-30T06:42:38.139ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
69572025-04-30T06:42:38.185ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37794e00-66bb-4eb0-8095-1b89fc38f323", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69582025-04-30T06:42:38.185ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
69592025-04-30T06:42:38.190ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16f1e979-8e25-43b3-a3d7-793fd8e8eac9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69602025-04-30T06:42:38.190ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
69612025-04-30T06:42:38.192ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afe4cc14-6621-43d6-b192-f41557db2e02", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69622025-04-30T06:42:38.193ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
69632025-04-30T06:42:38.196ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
69642025-04-30T06:42:38.201ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61a813fa-f681-4bca-bc97-df4b1f75f4bc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69652025-04-30T06:42:38.201ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
69662025-04-30T06:42:38.205ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "067e4add-c347-4e10-9793-745b25208cd2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69672025-04-30T06:42:38.206ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
69682025-04-30T06:42:38.212ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6cd762c-7163-410f-9888-427a8f1b78b1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69692025-04-30T06:42:38.212ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
69702025-04-30T06:42:38.214ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3979838-1aff-4c86-9741-a6a2d0e380f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69712025-04-30T06:42:38.214ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
69722025-04-30T06:42:38.215ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a5d9eff-4dd5-400e-afb3-99feba347fd0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69732025-04-30T06:42:38.215ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
69742025-04-30T06:42:38.218ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a5549f0-1f87-4a3a-9378-ca0ffc26832f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69752025-04-30T06:42:38.219ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
69762025-04-30T06:42:38.223ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdebcb3a-2701-403e-9711-42376f06dcab", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69772025-04-30T06:42:38.223ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
69782025-04-30T06:42:38.225ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5593dc3-8362-47ff-9698-510cd2135e32", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69792025-04-30T06:42:38.225ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
69802025-04-30T06:42:38.228ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0a05861-d4d5-42d9-9dac-2347a55b3300", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69812025-04-30T06:42:38.228ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
69822025-04-30T06:42:38.231ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41e22379-3a0c-46b1-a037-319be9223b9f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69832025-04-30T06:42:38.231ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
69842025-04-30T06:42:38.234ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f7ae35d-7c7c-4b8e-a826-1c7c196a1b91", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69852025-04-30T06:42:38.234ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
69862025-04-30T06:42:38.238ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1321d494-8a62-47fa-8d3e-6aa04ebcaf49", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69872025-04-30T06:42:38.238ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
69882025-04-30T06:42:38.242ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5ba47e3-463e-4f5d-aabf-2786057803cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69892025-04-30T06:42:38.243ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
69902025-04-30T06:42:38.247ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "542579f9-89db-4412-aade-b2f61de75a6c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69912025-04-30T06:42:38.247ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
69922025-04-30T06:42:38.247ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69932025-04-30T06:42:38.247ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69942025-04-30T06:42:38.247ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69952025-04-30T06:42:38.247ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
69962025-04-30T06:42:38.248ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69972025-04-30T06:42:38.249ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "760881bf-95db-474d-82e6-abdc33e86c44", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
69982025-04-30T06:42:38.249ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
69992025-04-30T06:42:38.251ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93750aec-1eb3-4c07-bc1f-3d7b2d766e77", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70002025-04-30T06:42:38.251ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
70012025-04-30T06:42:38.252ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70022025-04-30T06:42:38.260ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70032025-04-30T06:42:38.260ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70042025-04-30T06:42:38.260ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
70052025-04-30T06:42:38.261ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70062025-04-30T06:42:38.264ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b5ac7a6-b125-4e47-a51b-81387f6c324d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70072025-04-30T06:42:38.265ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
70082025-04-30T06:42:38.266ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70092025-04-30T06:42:38.266ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
70102025-04-30T06:42:38.267ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70112025-04-30T06:42:38.269ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cab2776b-054a-466d-a7f0-e4ade3626796", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70122025-04-30T06:42:38.269ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
70132025-04-30T06:42:38.272ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5e64f38-a685-424a-8f22-faa4e2b701ba", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70142025-04-30T06:42:38.272ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
70152025-04-30T06:42:38.280ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5eee332-00f4-4499-beee-c95e0c2eee13", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70162025-04-30T06:42:38.280ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
70172025-04-30T06:42:38.285ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d2ed9c8-4b0c-47cc-b866-e5daca057324", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70182025-04-30T06:42:38.285ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
70192025-04-30T06:42:38.295ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86ed75d0-56d3-4902-b734-0a0878eb6990", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70202025-04-30T06:42:38.296ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
70212025-04-30T06:42:38.314ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
70222025-04-30T06:42:38.318ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e99eeec-61ac-43cd-9bea-1ede8df8fa6d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70232025-04-30T06:42:38.319ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
70242025-04-30T06:42:38.322ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "511403a4-8ebe-4362-bea9-1fd955a32629", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70252025-04-30T06:42:38.322ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
70262025-04-30T06:42:38.325ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dff9c2d-9850-4a6b-9444-7130a70e8c0c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70272025-04-30T06:42:38.325ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
70282025-04-30T06:42:38.326ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a33446ab-d364-48c9-8fc3-de74f498e15c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70292025-04-30T06:42:38.327ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
70302025-04-30T06:42:38.328ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "462a6082-9418-42c6-816e-2e26d91018ac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70312025-04-30T06:42:38.328ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
70322025-04-30T06:42:38.329ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bd80407-d9a5-4460-95d8-ef388c8ef42b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70332025-04-30T06:42:38.330ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
70342025-04-30T06:42:38.330ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fdd2d54-5f50-490c-980b-84866a38687c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70352025-04-30T06:42:38.331ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
70362025-04-30T06:42:38.332ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f48f9255-aa73-4f05-b41b-88be010bfd63", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70372025-04-30T06:42:38.332ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
70382025-04-30T06:42:38.333ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e833c39-3aa3-48d1-8bf8-9eb3e3035c16", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70392025-04-30T06:42:38.334ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
70402025-04-30T06:42:38.336ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab19d6ec-0fc5-4f3b-b1c2-fb93412f7f14", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70412025-04-30T06:42:38.336ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
70422025-04-30T06:42:38.338ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d29c527f-a56a-4479-8f0c-2b8b38df2fd4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70432025-04-30T06:42:38.338ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
70442025-04-30T06:42:38.341ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f95b159b-9aad-47a3-adcc-a1c1bea28a49", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70452025-04-30T06:42:38.342ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
70462025-04-30T06:42:38.344ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a52dfec-8da5-42ef-927a-491c860543f1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70472025-04-30T06:42:38.344ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
70482025-04-30T06:42:38.348ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64acdd2a-e577-4076-9ac4-efee5b08e519", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70492025-04-30T06:42:38.348ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
70502025-04-30T06:42:38.349ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbb95f03-a48a-4d44-9b4a-d70b643b1c98", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70512025-04-30T06:42:38.349ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
70522025-04-30T06:42:38.350ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e68e9de5-2e18-4c28-951c-38b4b1cdb735", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70532025-04-30T06:42:38.350ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
70542025-04-30T06:42:38.351ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3f31c65-7ea9-4c4c-ae42-bc80b61e9937", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70552025-04-30T06:42:38.352ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
70562025-04-30T06:42:38.352ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "937f65ae-a275-4167-93a3-1af6ea30cf15", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70572025-04-30T06:42:38.353ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70582025-04-30T06:42:38.353ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70592025-04-30T06:42:38.353ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70602025-04-30T06:42:38.353ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70612025-04-30T06:42:38.353ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70622025-04-30T06:42:38.356ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70632025-04-30T06:42:38.361ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70642025-04-30T06:42:38.361ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70652025-04-30T06:42:38.361ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70662025-04-30T06:42:38.361ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70672025-04-30T06:42:38.361ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70682025-04-30T06:42:38.362ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70692025-04-30T06:42:38.366ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70702025-04-30T06:42:38.366ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70712025-04-30T06:42:38.366ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70722025-04-30T06:42:38.366ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
70732025-04-30T06:42:38.367ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70742025-04-30T06:42:38.368ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70752025-04-30T06:42:38.371ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70762025-04-30T06:42:38.372ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70772025-04-30T06:42:38.372ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70782025-04-30T06:42:38.373ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70792025-04-30T06:42:38.377ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70802025-04-30T06:42:38.383ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70812025-04-30T06:42:38.383ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70822025-04-30T06:42:38.383ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
70832025-04-30T06:42:38.384ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70842025-04-30T06:42:38.454ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
70852025-04-30T06:42:38.454ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
70862025-04-30T06:42:38.455ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:60895 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
70872025-04-30T06:42:38.456ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:60895 req_id = 7af4fb41-2ec6-4065-887c-f660b393eed3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70882025-04-30T06:42:38.457ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1685 local_addr = [::1]:62607 method = GET remote_addr = [::1]:60895 req_id = 7af4fb41-2ec6-4065-887c-f660b393eed3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70892025-04-30T06:42:38.458ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70902025-04-30T06:42:38.458ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
70912025-04-30T06:42:38.459ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70922025-04-30T06:42:38.460ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7af4fb41-2ec6-4065-887c-f660b393eed3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
70932025-04-30T06:42:38.469ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70942025-04-30T06:42:38.469ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70952025-04-30T06:42:38.469ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
70962025-04-30T06:42:38.475ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70972025-04-30T06:42:38.475ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70982025-04-30T06:42:38.475ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70992025-04-30T06:42:38.475ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71002025-04-30T06:42:38.477ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71012025-04-30T06:42:38.482ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71022025-04-30T06:42:38.487ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
71032025-04-30T06:42:38.487ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71042025-04-30T06:42:38.487ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71052025-04-30T06:42:38.488ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71062025-04-30T06:42:38.520ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
71072025-04-30T06:42:38.533ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
71082025-04-30T06:42:38.534ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
71092025-04-30T06:42:38.534ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 646.011446ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
71102025-04-30T06:42:38.534ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
71112025-04-30T06:42:38.534ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
71122025-04-30T06:42:38.534ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71132025-04-30T06:42:38.534ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71142025-04-30T06:42:38.534ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71152025-04-30T06:42:38.535ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71162025-04-30T06:42:38.538ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71172025-04-30T06:42:38.543ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
71182025-04-30T06:42:38.543ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71192025-04-30T06:42:38.543ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71202025-04-30T06:42:38.544ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71212025-04-30T06:42:38.552ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71222025-04-30T06:42:38.553ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71232025-04-30T06:42:38.553ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
71242025-04-30T06:42:38.553ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
71252025-04-30T06:42:38.556ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ae 51 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .Q..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52818 req_id = 8cade83e-0b36-4f67-9c1b-e8678826088b
71262025-04-30T06:42:38.557ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44625,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52818 req_id = 8cade83e-0b36-4f67-9c1b-e8678826088b
71272025-04-30T06:42:38.557ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
71282025-04-30T06:42:38.558ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71292025-04-30T06:42:38.558ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52818 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8cade83e-0b36-4f67-9c1b-e8678826088b
71302025-04-30T06:42:38.558ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71312025-04-30T06:42:38.559ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71322025-04-30T06:42:38.559ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71332025-04-30T06:42:38.559ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71342025-04-30T06:42:38.559ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4223, tv_nsec: 358719603 } })
71352025-04-30T06:42:38.559ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
71362025-04-30T06:42:38.560ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 76 78 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e vx..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56942 req_id = 35a1d144-2cd0-4b62-8782-5dfd304a27fb
71372025-04-30T06:42:38.560ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30328,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56942 req_id = 35a1d144-2cd0-4b62-8782-5dfd304a27fb
71382025-04-30T06:42:38.560ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71392025-04-30T06:42:38.560ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56942 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 35a1d144-2cd0-4b62-8782-5dfd304a27fb
71402025-04-30T06:42:38.684ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
71412025-04-30T06:42:38.685ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71422025-04-30T06:42:38.685ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71432025-04-30T06:42:38.685ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71442025-04-30T06:42:38.685ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
71452025-04-30T06:42:38.686ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71462025-04-30T06:42:38.686ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 00 1e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62056 req_id = b6a1c13b-544a-4e62-97c9-8577fa6e134a
71472025-04-30T06:42:38.686ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62056 req_id = b6a1c13b-544a-4e62-97c9-8577fa6e134a
71482025-04-30T06:42:38.687ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
71492025-04-30T06:42:38.687ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62056 req_id = b6a1c13b-544a-4e62-97c9-8577fa6e134a
71502025-04-30T06:42:38.688ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
71512025-04-30T06:42:38.688ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
71522025-04-30T06:42:38.688ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
71532025-04-30T06:42:38.689ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71542025-04-30T06:42:38.724ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71552025-04-30T06:42:38.724ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71562025-04-30T06:42:38.724ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
71572025-04-30T06:42:38.725ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71582025-04-30T06:42:38.731ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71592025-04-30T06:42:38.731ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
71602025-04-30T06:42:38.731ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71612025-04-30T06:42:38.732ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a43e99c-ad65-4aed-9fdb-334144da38c3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71622025-04-30T06:42:38.732ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
71632025-04-30T06:42:38.736ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9958822b-0a0b-4f4b-bf44-8a67eec15a14", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71642025-04-30T06:42:38.736ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
71652025-04-30T06:42:38.745ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "326e4d47-d15a-42ca-bfef-464ccebb3430", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71662025-04-30T06:42:38.746ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
71672025-04-30T06:42:38.751ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9447445-90b2-46ba-b152-76b9144cbec9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71682025-04-30T06:42:38.752ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
71692025-04-30T06:42:38.787ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db7becc2-a051-444f-8611-3b9c150c24f2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71702025-04-30T06:42:38.787ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
71712025-04-30T06:42:38.790ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ae5379-f31d-49f5-8f5d-d806de38b046", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71722025-04-30T06:42:38.791ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
71732025-04-30T06:42:38.794ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
71742025-04-30T06:42:38.796ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92b7f755-1543-4925-a286-d79ca4bda952", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71752025-04-30T06:42:38.796ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
71762025-04-30T06:42:38.799ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c0714a7-caec-497c-a381-27be47b457d9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71772025-04-30T06:42:38.800ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
71782025-04-30T06:42:38.801ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b33b265b-7cba-4f9a-94b1-5ed3546e847a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71792025-04-30T06:42:38.802ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
71802025-04-30T06:42:38.803ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81616402-7e05-4b9c-bf6e-c708910fce6a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71812025-04-30T06:42:38.803ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
71822025-04-30T06:42:38.804ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2dff2df-b253-4441-b011-a06c2c12577a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71832025-04-30T06:42:38.805ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
71842025-04-30T06:42:38.807ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59254720-7eaf-4375-a1fe-03fb2aec6abb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:37 GMT"} })
71852025-04-30T06:42:38.807ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
71862025-04-30T06:42:38.809ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "118eb3f8-a755-41bc-a01d-0512e504329d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
71872025-04-30T06:42:38.809ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
71882025-04-30T06:42:38.812ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb6d7fe4-d434-4101-8a78-3f5fe27d20f1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
71892025-04-30T06:42:38.813ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
71902025-04-30T06:42:38.825ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "719723f2-0acb-4b2d-bcac-e2cb909cf5c6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
71912025-04-30T06:42:38.826ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
71922025-04-30T06:42:38.830ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2da49e8-8b65-42ad-8d51-79d2042b1213", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
71932025-04-30T06:42:38.846ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
71942025-04-30T06:42:38.847ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71952025-04-30T06:42:38.847ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71962025-04-30T06:42:38.856ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71972025-04-30T06:42:38.856ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
71982025-04-30T06:42:38.856ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71992025-04-30T06:42:38.860ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72002025-04-30T06:42:38.866ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72012025-04-30T06:42:38.866ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72022025-04-30T06:42:38.866ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
72032025-04-30T06:42:38.866ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72042025-04-30T06:42:38.867ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee239900-6303-4149-82a7-98e0de6142ba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72052025-04-30T06:42:38.867ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
72062025-04-30T06:42:38.869ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72072025-04-30T06:42:38.869ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
72082025-04-30T06:42:38.870ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72092025-04-30T06:42:38.873ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fbe5535-867c-438e-9365-37df42d0cab1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72102025-04-30T06:42:38.873ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
72112025-04-30T06:42:38.878ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bed0d16e-eedd-4645-8a71-08e3e4e25ba5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72122025-04-30T06:42:38.879ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
72132025-04-30T06:42:38.891ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "602bc9ff-b61b-4c6e-8b50-72aeed028cb9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72142025-04-30T06:42:38.891ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
72152025-04-30T06:42:38.894ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25d3d4d4-d78b-47ee-9aff-72be607bd77b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72162025-04-30T06:42:38.894ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
72172025-04-30T06:42:38.897ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "549a4962-d672-4739-9d23-422cbac46979", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72182025-04-30T06:42:38.897ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
72192025-04-30T06:42:38.900ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb27bb19-a892-4f30-b522-33d28ed7a43e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72202025-04-30T06:42:38.900ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
72212025-04-30T06:42:38.903ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
72222025-04-30T06:42:38.903ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
72232025-04-30T06:42:38.903ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
72242025-04-30T06:42:38.904ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
72252025-04-30T06:42:38.915ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
72262025-04-30T06:42:38.915ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
72272025-04-30T06:42:38.920ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
72282025-04-30T06:42:38.920ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
72292025-04-30T06:42:38.920ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e149d7fc-038a-4f56-8526-becfa9c0b66e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72302025-04-30T06:42:38.920ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
72312025-04-30T06:42:38.965ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
72322025-04-30T06:42:38.967ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b5baf76-2149-43c5-b37f-1d72422d0198", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72332025-04-30T06:42:38.968ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
72342025-04-30T06:42:38.970ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc09ff59-eea7-48b7-8284-8d1adc9652aa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72352025-04-30T06:42:38.970ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
72362025-04-30T06:42:38.972ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40fcea7b-a1a6-4bbc-9db5-7bc2774cfe2b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72372025-04-30T06:42:38.972ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
72382025-04-30T06:42:38.975ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e566f2c4-f362-46b6-8254-6374ecce174a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72392025-04-30T06:42:38.976ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
72402025-04-30T06:42:38.978ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcc04419-6e28-4645-87cf-d0a2d55a0a14", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72412025-04-30T06:42:38.978ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
72422025-04-30T06:42:38.986ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "352203f1-4807-48c7-917c-fd2454790356", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72432025-04-30T06:42:38.990ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
72442025-04-30T06:42:38.994ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a183cfb-0596-4e3c-8319-896b298efd26", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72452025-04-30T06:42:38.994ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
72462025-04-30T06:42:39.002ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b2d1500-1572-4654-b591-03cab1f8734f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72472025-04-30T06:42:39.002ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
72482025-04-30T06:42:39.003ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cab4ad8-379f-4c4a-bbde-24c7174cedba", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72492025-04-30T06:42:39.003ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
72502025-04-30T06:42:39.004ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
72512025-04-30T06:42:39.004ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94122d0f-147e-4118-8caf-25170b376261", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72522025-04-30T06:42:39.004ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
72532025-04-30T06:42:39.008ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1dd5553-ad71-40b2-9cad-c1be9df47e76", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72542025-04-30T06:42:39.008ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
72552025-04-30T06:42:39.012ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb1adb4c-8079-42b8-97ec-147206357aed", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72562025-04-30T06:42:39.013ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
72572025-04-30T06:42:39.015ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97b2115c-e88d-45af-b4f8-bb7db512bc64", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72582025-04-30T06:42:39.015ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
72592025-04-30T06:42:39.017ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72602025-04-30T06:42:39.017ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72612025-04-30T06:42:39.017ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72622025-04-30T06:42:39.017ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
72632025-04-30T06:42:39.017ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72642025-04-30T06:42:39.018ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "542f445d-0b85-496f-9bc1-5863d69a5e57", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72652025-04-30T06:42:39.018ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
72662025-04-30T06:42:39.038ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72672025-04-30T06:42:39.045ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72682025-04-30T06:42:39.045ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72692025-04-30T06:42:39.045ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
72702025-04-30T06:42:39.045ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72712025-04-30T06:42:39.045ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "588a44cc-19d6-4dca-9e6f-b86ed400c18e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72722025-04-30T06:42:39.046ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
72732025-04-30T06:42:39.054ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72742025-04-30T06:42:39.054ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
72752025-04-30T06:42:39.055ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72762025-04-30T06:42:39.056ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92bd8f02-4584-422e-9c61-7e3c6ea67dfd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72772025-04-30T06:42:39.056ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
72782025-04-30T06:42:39.059ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49177a6b-0e42-492f-a262-b4efdd22e47c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72792025-04-30T06:42:39.059ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
72802025-04-30T06:42:39.060ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fde22b85-9d28-4570-b043-f7136d673165", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72812025-04-30T06:42:39.060ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
72822025-04-30T06:42:39.062ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b759cc4-4812-4743-82ac-f24e70cf7fc1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72832025-04-30T06:42:39.062ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
72842025-04-30T06:42:39.072ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f985f8cb-68b6-4ccc-bd1d-b9101745cc59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72852025-04-30T06:42:39.072ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
72862025-04-30T06:42:39.097ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da9f6700-5255-401e-bdad-323de9db1c24", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
72872025-04-30T06:42:39.098ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72882025-04-30T06:42:39.098ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72892025-04-30T06:42:39.098ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72902025-04-30T06:42:39.098ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72912025-04-30T06:42:39.098ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72922025-04-30T06:42:39.108ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72932025-04-30T06:42:39.111ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72942025-04-30T06:42:39.111ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72952025-04-30T06:42:39.112ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72962025-04-30T06:42:39.112ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72972025-04-30T06:42:39.112ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72982025-04-30T06:42:39.112ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72992025-04-30T06:42:39.115ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73002025-04-30T06:42:39.119ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
73012025-04-30T06:42:39.119ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73022025-04-30T06:42:39.119ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73032025-04-30T06:42:39.120ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73042025-04-30T06:42:39.222ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
73052025-04-30T06:42:39.223ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
73062025-04-30T06:42:39.228ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:55005 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
73072025-04-30T06:42:39.229ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:55005 req_id = b385437c-523d-46e4-8a7b-c034bbcf5761 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73082025-04-30T06:42:39.232ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 2535 local_addr = [::1]:62607 method = GET remote_addr = [::1]:55005 req_id = b385437c-523d-46e4-8a7b-c034bbcf5761 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73092025-04-30T06:42:39.232ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b385437c-523d-46e4-8a7b-c034bbcf5761", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
73102025-04-30T06:42:39.236ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73112025-04-30T06:42:39.236ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73122025-04-30T06:42:39.236ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
73132025-04-30T06:42:39.239ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73142025-04-30T06:42:39.239ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73152025-04-30T06:42:39.239ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73162025-04-30T06:42:39.239ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73172025-04-30T06:42:39.240ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73182025-04-30T06:42:39.245ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73192025-04-30T06:42:39.250ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73202025-04-30T06:42:39.252ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73212025-04-30T06:42:39.252ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73222025-04-30T06:42:39.253ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73232025-04-30T06:42:39.268ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
73242025-04-30T06:42:39.320ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73252025-04-30T06:42:39.320ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73262025-04-30T06:42:39.320ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73272025-04-30T06:42:39.320ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
73282025-04-30T06:42:39.321ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73292025-04-30T06:42:39.323ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73302025-04-30T06:42:39.330ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73312025-04-30T06:42:39.331ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73322025-04-30T06:42:39.331ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
73332025-04-30T06:42:39.332ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73342025-04-30T06:42:39.334ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
73352025-04-30T06:42:39.334ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
73362025-04-30T06:42:39.334ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 800.048204ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
73372025-04-30T06:42:39.334ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
73382025-04-30T06:42:39.334ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73392025-04-30T06:42:39.334ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73402025-04-30T06:42:39.334ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73412025-04-30T06:42:39.334ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73422025-04-30T06:42:39.336ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73432025-04-30T06:42:39.337ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73442025-04-30T06:42:39.337ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
73452025-04-30T06:42:39.338ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73462025-04-30T06:42:39.338ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73472025-04-30T06:42:39.343ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73482025-04-30T06:42:39.343ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73492025-04-30T06:42:39.343ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73502025-04-30T06:42:39.344ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73512025-04-30T06:42:39.352ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
73522025-04-30T06:42:39.352ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
73532025-04-30T06:42:39.353ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e3 0a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61568 req_id = 3fc5d6a1-4198-4160-8a51-4b715c5e7d85
73542025-04-30T06:42:39.353ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58122,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61568 req_id = 3fc5d6a1-4198-4160-8a51-4b715c5e7d85
73552025-04-30T06:42:39.354ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
73562025-04-30T06:42:39.354ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73572025-04-30T06:42:39.354ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61568 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3fc5d6a1-4198-4160-8a51-4b715c5e7d85
73582025-04-30T06:42:39.356ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4224, tv_nsec: 154149456 } })
73592025-04-30T06:42:39.357ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
73602025-04-30T06:42:39.358ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 00 73 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .s..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50757 req_id = f12b30d1-69f2-458e-976c-034940ae6d5b
73612025-04-30T06:42:39.358ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 115,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50757 req_id = f12b30d1-69f2-458e-976c-034940ae6d5b
73622025-04-30T06:42:39.358ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73632025-04-30T06:42:39.358ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50757 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f12b30d1-69f2-458e-976c-034940ae6d5b
73642025-04-30T06:42:39.451ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
73652025-04-30T06:42:39.461ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a3 4b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .K..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60382 req_id = 295c8aa6-d9f5-4ee3-b211-adfe8b52f647
73662025-04-30T06:42:39.462ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41803,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60382 req_id = 295c8aa6-d9f5-4ee3-b211-adfe8b52f647
73672025-04-30T06:42:39.463ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
73682025-04-30T06:42:39.463ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41803, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60382 req_id = 295c8aa6-d9f5-4ee3-b211-adfe8b52f647
73692025-04-30T06:42:39.464ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
73702025-04-30T06:42:39.464ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
73712025-04-30T06:42:39.464ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
73722025-04-30T06:42:39.466ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8e07d17-eb20-4077-8578-3e96715b84dd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
73732025-04-30T06:42:39.467ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
73742025-04-30T06:42:39.468ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5798c62-0777-42d0-a805-35e1bdca2450", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
73752025-04-30T06:42:39.468ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
73762025-04-30T06:42:39.473ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f109048-a2da-43af-83d5-286cacc96bca", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
73772025-04-30T06:42:39.473ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
73782025-04-30T06:42:39.475ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8db08b9-1066-4f84-aaa1-0da618322579", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
73792025-04-30T06:42:39.475ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
73802025-04-30T06:42:39.478ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0660fca2-b6c7-456d-a8b2-ab4517484e32", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
73812025-04-30T06:42:39.478ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
73822025-04-30T06:42:39.482ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dc3fc23-3e7a-4c55-bbc5-6f8153c4f2bc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
73832025-04-30T06:42:39.483ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
73842025-04-30T06:42:39.487ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
73852025-04-30T06:42:39.490ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
73862025-04-30T06:42:39.490ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73872025-04-30T06:42:39.490ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1e389dc8-0ac7-4a54-ae65-285cd360ccf1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73882025-04-30T06:42:39.490ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1e389dc8-0ac7-4a54-ae65-285cd360ccf1 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73892025-04-30T06:42:39.490ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1e389dc8-0ac7-4a54-ae65-285cd360ccf1 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73902025-04-30T06:42:39.491ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1e389dc8-0ac7-4a54-ae65-285cd360ccf1 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73912025-04-30T06:42:39.493ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb4b3e06-4a68-4e84-94cf-d26443427b77", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
73922025-04-30T06:42:39.493ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
73932025-04-30T06:42:39.494ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 4840 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1e389dc8-0ac7-4a54-ae65-285cd360ccf1 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
73942025-04-30T06:42:39.497ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e389dc8-0ac7-4a54-ae65-285cd360ccf1", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
73952025-04-30T06:42:39.498ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
73962025-04-30T06:42:39.498ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 01f69f68-686e-4ca2-b114-c533e37daec7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
73972025-04-30T06:42:39.508ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 01f69f68-686e-4ca2-b114-c533e37daec7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
73982025-04-30T06:42:39.508ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 01f69f68-686e-4ca2-b114-c533e37daec7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
73992025-04-30T06:42:39.509ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 01f69f68-686e-4ca2-b114-c533e37daec7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
74002025-04-30T06:42:39.520ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 21751 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 01f69f68-686e-4ca2-b114-c533e37daec7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
74012025-04-30T06:42:39.521ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "01f69f68-686e-4ca2-b114-c533e37daec7", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74022025-04-30T06:42:39.521ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74032025-04-30T06:42:39.521ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
74042025-04-30T06:42:39.521ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
74052025-04-30T06:42:39.521ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d90e18d5-5ef5-4bb7-bdd4-033822be3866", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74062025-04-30T06:42:39.522ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
74072025-04-30T06:42:39.525ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25445cf3-4c63-402d-a546-6c20a1ff64f7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74082025-04-30T06:42:39.525ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
74092025-04-30T06:42:39.527ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1124daea-2720-40c9-83d4-e71e72ec83a2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74102025-04-30T06:42:39.527ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
74112025-04-30T06:42:39.529ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fea016a-3b4b-45e8-9b25-f67cc731a394", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74122025-04-30T06:42:39.529ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
74132025-04-30T06:42:39.531ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e50c84d-9bb8-407f-a297-d295a44a50b4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74142025-04-30T06:42:39.532ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
74152025-04-30T06:42:39.535ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85b6e786-6e31-44fe-8392-3328799e125a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74162025-04-30T06:42:39.535ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
74172025-04-30T06:42:39.564ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74182025-04-30T06:42:39.564ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74192025-04-30T06:42:39.564ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74202025-04-30T06:42:39.564ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
74212025-04-30T06:42:39.565ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74222025-04-30T06:42:39.566ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74232025-04-30T06:42:39.566ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74242025-04-30T06:42:39.567ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4810db3-b7d2-4370-b98b-e9150c1e34e2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74252025-04-30T06:42:39.567ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
74262025-04-30T06:42:39.569ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74272025-04-30T06:42:39.575ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74282025-04-30T06:42:39.575ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74292025-04-30T06:42:39.575ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
74302025-04-30T06:42:39.575ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74312025-04-30T06:42:39.576ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74322025-04-30T06:42:39.577ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74332025-04-30T06:42:39.577ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74342025-04-30T06:42:39.577ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
74352025-04-30T06:42:39.581ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74362025-04-30T06:42:39.581ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
74372025-04-30T06:42:39.581ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74382025-04-30T06:42:39.583ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a58c7f-ca36-4aaf-8b72-05b00d13bef6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74392025-04-30T06:42:39.584ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
74402025-04-30T06:42:39.588ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b04b081f-32ed-44e7-ad38-112369509400", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74412025-04-30T06:42:39.588ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
74422025-04-30T06:42:39.596ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c49a811-a3d1-48ff-8f3b-5ced91530de5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74432025-04-30T06:42:39.596ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
74442025-04-30T06:42:39.602ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1674233-627a-4dfc-8119-b94bb094f9f4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74452025-04-30T06:42:39.603ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
74462025-04-30T06:42:39.607ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3ab2e3f-d426-4e0f-b74b-5439eb20231b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74472025-04-30T06:42:39.608ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
74482025-04-30T06:42:39.609ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22f2a440-4afc-471e-aa09-5ef7aae76dc8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74492025-04-30T06:42:39.610ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
74502025-04-30T06:42:39.611ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "433e989f-d567-457b-ba3c-d3aba73c08dd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74512025-04-30T06:42:39.612ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
74522025-04-30T06:42:39.615ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8586d3d-2f4a-47da-9b0a-d0edc2d1d837", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74532025-04-30T06:42:39.615ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
74542025-04-30T06:42:39.616ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
74552025-04-30T06:42:39.618ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d56f64d0-13d7-4992-b3fd-8bcebab7f8c8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74562025-04-30T06:42:39.618ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
74572025-04-30T06:42:39.621ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a78a1ff2-48c1-4e7b-bcb4-15013b40ecbb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74582025-04-30T06:42:39.621ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
74592025-04-30T06:42:39.625ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5215e23a-37ff-4697-9700-c0d927534fb2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74602025-04-30T06:42:39.626ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
74612025-04-30T06:42:39.627ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
74622025-04-30T06:42:39.627ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
74632025-04-30T06:42:39.628ZINFOtest_delete_snapshot (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:65039 remote_addr = 127.0.0.1:55281
74642025-04-30T06:42:39.628ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = 127.0.0.1:65039 method = GET remote_addr = 127.0.0.1:55281 req_id = bd533794-01ab-4a78-9017-cb95697fccf8 uri = /de3c7c83-8252-43f4-ba2e-cbd92004c4b0
74652025-04-30T06:42:39.635ZINFOtest_delete_snapshot (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 6764 local_addr = 127.0.0.1:65039 method = GET remote_addr = 127.0.0.1:55281 req_id = bd533794-01ab-4a78-9017-cb95697fccf8 response_code = 200 uri = /de3c7c83-8252-43f4-ba2e-cbd92004c4b0
74662025-04-30T06:42:39.635ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
74672025-04-30T06:42:39.652ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
74682025-04-30T06:42:39.652ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
74692025-04-30T06:42:39.653ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
74702025-04-30T06:42:39.653ZDEBGtest_delete_snapshot (results-sink): inserting 18 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74712025-04-30T06:42:39.653ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 18 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74722025-04-30T06:42:39.653ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 18 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
74732025-04-30T06:42:39.655ZDEBGtest_delete_snapshot (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
74742025-04-30T06:42:39.655ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
74752025-04-30T06:42:39.657ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb82981d-44b5-4770-bf24-4af83f4a9f14", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74762025-04-30T06:42:39.657ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
74772025-04-30T06:42:39.660ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e47c67da-fd1c-43d7-b2d3-64482cd0af10", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
74782025-04-30T06:42:39.660ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
74792025-04-30T06:42:39.663ZTRACtest_delete_snapshot (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
74802025-04-30T06:42:39.663ZDEBGtest_delete_snapshot (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:39.655208889Z } timeseries_name = http_service:request_latency_histogram
74812025-04-30T06:42:39.670ZDEBGtest_delete_snapshot (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
74822025-04-30T06:42:39.670ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
74832025-04-30T06:42:39.670ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74842025-04-30T06:42:39.670ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74852025-04-30T06:42:39.670ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74862025-04-30T06:42:39.671ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
74872025-04-30T06:42:39.672ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74882025-04-30T06:42:39.673ZTRACtest_delete_snapshot (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
74892025-04-30T06:42:39.673ZDEBGtest_delete_snapshot (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:39.670360708Z } timeseries_name = collection_target:cpus_provisioned
74902025-04-30T06:42:39.674ZDEBGtest_delete_snapshot (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
74912025-04-30T06:42:39.674ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
74922025-04-30T06:42:39.677ZTRACtest_delete_snapshot (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
74932025-04-30T06:42:39.677ZDEBGtest_delete_snapshot (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:39.674720555Z } timeseries_name = collection_target:ram_provisioned
74942025-04-30T06:42:39.678ZDEBGtest_delete_snapshot (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
74952025-04-30T06:42:39.678ZTRACtest_delete_snapshot (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
74962025-04-30T06:42:39.679ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74972025-04-30T06:42:39.686ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74982025-04-30T06:42:39.686ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74992025-04-30T06:42:39.687ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
75002025-04-30T06:42:39.687ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75012025-04-30T06:42:39.699ZTRACtest_delete_snapshot (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
75022025-04-30T06:42:39.699ZDEBGtest_delete_snapshot (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:42:39.678775113Z } timeseries_name = collection_target:virtual_disk_space_provisioned
75032025-04-30T06:42:39.709ZDEBGtest_delete_snapshot (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
75042025-04-30T06:42:39.709ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
75052025-04-30T06:42:39.709ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e724546f-6c2d-484f-a8a6-a4f0005bddb6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75062025-04-30T06:42:39.709ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
75072025-04-30T06:42:39.710ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75082025-04-30T06:42:39.710ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
75092025-04-30T06:42:39.710ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75102025-04-30T06:42:39.715ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
75112025-04-30T06:42:39.718ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac99c47-8087-4077-b5a8-f883d95afff1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75122025-04-30T06:42:39.718ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
75132025-04-30T06:42:39.723ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 12 table_name = fields_string
75142025-04-30T06:42:39.723ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
75152025-04-30T06:42:39.729ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 5 table_name = fields_u16
75162025-04-30T06:42:39.729ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
75172025-04-30T06:42:39.729ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58ad1a32-e4cc-483e-baab-5433a4ad44c7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75182025-04-30T06:42:39.729ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
75192025-04-30T06:42:39.739ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5a7f99e-a4da-472e-9e1d-a27da4468265", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75202025-04-30T06:42:39.740ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
75212025-04-30T06:42:39.741ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 17 table_name = fields_uuid
75222025-04-30T06:42:39.741ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
75232025-04-30T06:42:39.741ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30715ca3-bca9-4105-9c03-c103ed9f79a8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75242025-04-30T06:42:39.741ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
75252025-04-30T06:42:39.743ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c51e3fe5-9c21-49dc-9e63-58a6e4b6bd3e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75262025-04-30T06:42:39.743ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
75272025-04-30T06:42:39.745ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "867e8e12-66b2-4b98-ad0e-d21726c9a749", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75282025-04-30T06:42:39.745ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
75292025-04-30T06:42:39.746ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b336cc8-2dec-431e-be46-cdb7b9560c35", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75302025-04-30T06:42:39.746ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
75312025-04-30T06:42:39.749ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56e03b58-8a8e-4939-9ce6-9312db1b0f4b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75322025-04-30T06:42:39.749ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
75332025-04-30T06:42:39.752ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 5 table_name = measurements_histogramu64
75342025-04-30T06:42:39.752ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 13 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
75352025-04-30T06:42:39.753ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
75362025-04-30T06:42:39.754ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
75372025-04-30T06:42:39.761ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 13 table_name = measurements_i64
75382025-04-30T06:42:39.761ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75392025-04-30T06:42:39.762ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e6f6b8d-dd0e-47ae-91d2-d5c98d08968b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75402025-04-30T06:42:39.762ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
75412025-04-30T06:42:39.765ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91a55c2c-15c7-4405-887c-d7599d5637ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75422025-04-30T06:42:39.765ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
75432025-04-30T06:42:39.767ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "032351d1-199b-455c-b85a-02f78806c3a5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75442025-04-30T06:42:39.768ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
75452025-04-30T06:42:39.770ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e94198f-c7f2-4106-b1c9-c0da67e2d319", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75462025-04-30T06:42:39.770ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
75472025-04-30T06:42:39.772ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b25e7224-845b-4efc-97c5-1faa5e3a5426", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75482025-04-30T06:42:39.772ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
75492025-04-30T06:42:39.773ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4450fb50-8913-48b8-907b-ffb1f94552f3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75502025-04-30T06:42:39.773ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
75512025-04-30T06:42:39.790ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fd24e87-c446-467d-9062-2e5991f16db9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75522025-04-30T06:42:39.790ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
75532025-04-30T06:42:39.791ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d17d414f-995e-4733-bc64-0e93ad3238df", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75542025-04-30T06:42:39.794ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
75552025-04-30T06:42:39.795ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cf884c2-9dc3-40f4-bb7a-f0664e207a6b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:38 GMT"} })
75562025-04-30T06:42:39.795ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
75572025-04-30T06:42:39.795ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75582025-04-30T06:42:39.797ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75592025-04-30T06:42:39.797ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75602025-04-30T06:42:39.798ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75612025-04-30T06:42:39.806ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75622025-04-30T06:42:39.806ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75632025-04-30T06:42:39.806ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75642025-04-30T06:42:39.806ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
75652025-04-30T06:42:39.807ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75662025-04-30T06:42:39.810ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75672025-04-30T06:42:39.814ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
75682025-04-30T06:42:39.814ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
75692025-04-30T06:42:39.814ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75702025-04-30T06:42:39.814ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75712025-04-30T06:42:39.814ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75722025-04-30T06:42:39.814ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75732025-04-30T06:42:39.830ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75742025-04-30T06:42:39.835ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75752025-04-30T06:42:39.836ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75762025-04-30T06:42:39.836ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
75772025-04-30T06:42:39.836ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75782025-04-30T06:42:39.837ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75792025-04-30T06:42:39.840ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
75802025-04-30T06:42:39.840ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75812025-04-30T06:42:39.840ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75822025-04-30T06:42:39.841ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75832025-04-30T06:42:39.851ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75842025-04-30T06:42:39.851ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
75852025-04-30T06:42:39.852ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75862025-04-30T06:42:39.908ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
75872025-04-30T06:42:39.909ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
75882025-04-30T06:42:39.910ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:41627 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
75892025-04-30T06:42:39.910ZTRACtest_delete_snapshot (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75902025-04-30T06:42:39.910ZTRACtest_delete_snapshot (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75912025-04-30T06:42:39.910ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:41627 req_id = 44e84dc2-1c94-42ef-adde-5983ca1b8b14 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75922025-04-30T06:42:39.910ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 592 local_addr = [::1]:62607 method = GET remote_addr = [::1]:41627 req_id = 44e84dc2-1c94-42ef-adde-5983ca1b8b14 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75932025-04-30T06:42:39.911ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "44e84dc2-1c94-42ef-adde-5983ca1b8b14", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
75942025-04-30T06:42:39.912ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75952025-04-30T06:42:39.912ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75962025-04-30T06:42:39.912ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
75972025-04-30T06:42:39.914ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
75982025-04-30T06:42:39.915ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75992025-04-30T06:42:39.915ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76002025-04-30T06:42:39.915ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76012025-04-30T06:42:39.916ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76022025-04-30T06:42:39.920ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76032025-04-30T06:42:39.934ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
76042025-04-30T06:42:39.934ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76052025-04-30T06:42:39.934ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76062025-04-30T06:42:39.934ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76072025-04-30T06:42:39.980ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
76082025-04-30T06:42:39.982ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
76092025-04-30T06:42:39.983ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
76102025-04-30T06:42:39.983ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 648.553892ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
76112025-04-30T06:42:39.983ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
76122025-04-30T06:42:39.983ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76132025-04-30T06:42:39.983ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76142025-04-30T06:42:39.983ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76152025-04-30T06:42:39.983ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76162025-04-30T06:42:39.985ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76172025-04-30T06:42:39.988ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76182025-04-30T06:42:39.992ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
76192025-04-30T06:42:39.992ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76202025-04-30T06:42:39.992ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76212025-04-30T06:42:39.993ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76222025-04-30T06:42:39.996ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
76232025-04-30T06:42:39.996ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
76242025-04-30T06:42:39.997ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 49 9b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 I............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59998 req_id = 41659087-5e0f-4c1c-8e47-7428c11fc474
76252025-04-30T06:42:39.997ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18843,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59998 req_id = 41659087-5e0f-4c1c-8e47-7428c11fc474
76262025-04-30T06:42:39.997ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
76272025-04-30T06:42:39.998ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
76282025-04-30T06:42:40.001ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59998 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 41659087-5e0f-4c1c-8e47-7428c11fc474
76292025-04-30T06:42:40.001ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4224, tv_nsec: 797829536 } })
76302025-04-30T06:42:40.001ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
76312025-04-30T06:42:40.001ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7d 64 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e }d..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49435 req_id = ad43c240-f314-4b32-a6ff-2f61b5ea1a26
76322025-04-30T06:42:40.001ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32100,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49435 req_id = ad43c240-f314-4b32-a6ff-2f61b5ea1a26
76332025-04-30T06:42:40.001ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
76342025-04-30T06:42:40.001ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49435 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ad43c240-f314-4b32-a6ff-2f61b5ea1a26
76352025-04-30T06:42:40.051ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
76362025-04-30T06:42:40.051ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76372025-04-30T06:42:40.051ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76382025-04-30T06:42:40.051ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76392025-04-30T06:42:40.051ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
76402025-04-30T06:42:40.052ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76412025-04-30T06:42:40.052ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b5 93 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64198 req_id = bb6db47b-d25f-42bd-9a32-4ee6c44b70fc
76422025-04-30T06:42:40.052ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46483,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64198 req_id = bb6db47b-d25f-42bd-9a32-4ee6c44b70fc
76432025-04-30T06:42:40.052ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
76442025-04-30T06:42:40.053ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46483, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64198 req_id = bb6db47b-d25f-42bd-9a32-4ee6c44b70fc
76452025-04-30T06:42:40.053ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
76462025-04-30T06:42:40.053ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
76472025-04-30T06:42:40.053ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
76482025-04-30T06:42:40.055ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76492025-04-30T06:42:40.084ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76502025-04-30T06:42:40.084ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76512025-04-30T06:42:40.084ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
76522025-04-30T06:42:40.085ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76532025-04-30T06:42:40.089ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee92fc13-d1a2-4472-bebe-db33eaf28986", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76542025-04-30T06:42:40.089ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
76552025-04-30T06:42:40.097ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76562025-04-30T06:42:40.097ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
76572025-04-30T06:42:40.098ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76582025-04-30T06:42:40.128ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c83349b-f688-47e7-916c-5dad08ebec54", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76592025-04-30T06:42:40.128ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
76602025-04-30T06:42:40.145ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a094af62-764c-40ae-9ba6-2a1d3ad428f4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76612025-04-30T06:42:40.146ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
76622025-04-30T06:42:40.148ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3354ca06-b3a3-4ed9-9f58-f4d5203d5119", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76632025-04-30T06:42:40.148ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
76642025-04-30T06:42:40.150ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba1ff9aa-6c68-44a9-ada6-77aae88d0b2d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76652025-04-30T06:42:40.150ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
76662025-04-30T06:42:40.152ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a939759-fe4e-4a1e-9e82-0acff313e0d0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76672025-04-30T06:42:40.152ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
76682025-04-30T06:42:40.154ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f91f9e7a-d933-4973-937e-663d7dd52ec5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76692025-04-30T06:42:40.155ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
76702025-04-30T06:42:40.157ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
76712025-04-30T06:42:40.159ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "446b0b15-cebd-4ba2-bcf4-eb6fe5b647b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76722025-04-30T06:42:40.159ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
76732025-04-30T06:42:40.160ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44d2d074-fc62-473d-9717-c09c5ab17cfa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76742025-04-30T06:42:40.160ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
76752025-04-30T06:42:40.161ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6780078-f7c3-4ead-9b3f-4e4a0a454282", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76762025-04-30T06:42:40.161ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
76772025-04-30T06:42:40.162ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f99df9-1441-4a10-9850-04ea978b5919", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76782025-04-30T06:42:40.162ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
76792025-04-30T06:42:40.163ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae558239-130f-4d25-bfa1-98fbc86072f2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76802025-04-30T06:42:40.163ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
76812025-04-30T06:42:40.164ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbdd7fe8-f507-42ac-9ce3-a00537ec4244", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76822025-04-30T06:42:40.164ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
76832025-04-30T06:42:40.166ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c83d74d-a6f4-4069-af9e-fbe1219e2b08", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76842025-04-30T06:42:40.166ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
76852025-04-30T06:42:40.168ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c5b259e-a5ce-4e87-bf94-cfb35156d45b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76862025-04-30T06:42:40.168ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
76872025-04-30T06:42:40.170ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0971ec20-63d2-4c68-aaa8-2bf2b15d05fb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76882025-04-30T06:42:40.170ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
76892025-04-30T06:42:40.172ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "479c7e5a-7888-406c-878f-194d6461d8d1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76902025-04-30T06:42:40.172ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
76912025-04-30T06:42:40.174ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89f0244e-de42-4cbe-8865-769c27b4814d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76922025-04-30T06:42:40.174ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
76932025-04-30T06:42:40.176ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1610493-0bbb-4028-91b8-d07ec227f757", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76942025-04-30T06:42:40.177ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
76952025-04-30T06:42:40.177ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5980dc92-ad59-40ad-95d9-d272cb834126", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76962025-04-30T06:42:40.178ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
76972025-04-30T06:42:40.179ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a20ff34b-1c4c-455e-9dfd-eab108938c21", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
76982025-04-30T06:42:40.179ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
76992025-04-30T06:42:40.180ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0323faa2-f2ac-4c7e-b519-3b5f72126133", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77002025-04-30T06:42:40.180ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
77012025-04-30T06:42:40.182ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6151f8b7-4cd4-432f-8d50-30f8a848317f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77022025-04-30T06:42:40.182ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
77032025-04-30T06:42:40.183ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dd0b989-98f5-4743-af45-1b8b347666c5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77042025-04-30T06:42:40.183ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
77052025-04-30T06:42:40.185ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76cde0e1-7c23-48a6-bf4b-656f47570ae7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77062025-04-30T06:42:40.185ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
77072025-04-30T06:42:40.188ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4592bd73-8c08-477c-b830-5b6d00bb7242", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77082025-04-30T06:42:40.188ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
77092025-04-30T06:42:40.191ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b96343-7e5d-42ce-b825-49f1df8f767b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77102025-04-30T06:42:40.191ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
77112025-04-30T06:42:40.193ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7840a886-0f1e-404d-b80e-aadde1e97c66", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77122025-04-30T06:42:40.193ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
77132025-04-30T06:42:40.197ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c239785-d504-4aab-a77b-40c6a27224e3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77142025-04-30T06:42:40.197ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
77152025-04-30T06:42:40.199ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d47f51d8-1625-4216-9c0c-44f15809ed01", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77162025-04-30T06:42:40.200ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
77172025-04-30T06:42:40.201ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d3225ce-8f7d-4a2d-90ff-9c7f07c137cc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77182025-04-30T06:42:40.202ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
77192025-04-30T06:42:40.203ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c7fafd-6c03-42e2-867b-9b45cd089f4c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77202025-04-30T06:42:40.204ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
77212025-04-30T06:42:40.205ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "753458f2-207e-46db-b7e5-d6bc3a1f8710", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77222025-04-30T06:42:40.206ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
77232025-04-30T06:42:40.209ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77242025-04-30T06:42:40.209ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77252025-04-30T06:42:40.209ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77262025-04-30T06:42:40.209ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
77272025-04-30T06:42:40.210ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77282025-04-30T06:42:40.211ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "703acef4-7e5d-426b-b568-b53cbd030bd5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77292025-04-30T06:42:40.211ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
77302025-04-30T06:42:40.214ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93a422d9-3149-4fb5-a928-5ea60a4dbdf3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77312025-04-30T06:42:40.214ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
77322025-04-30T06:42:40.214ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77332025-04-30T06:42:40.221ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77342025-04-30T06:42:40.221ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77352025-04-30T06:42:40.221ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
77362025-04-30T06:42:40.222ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77372025-04-30T06:42:40.226ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77382025-04-30T06:42:40.226ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
77392025-04-30T06:42:40.227ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77402025-04-30T06:42:40.229ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c3f3339-2853-4c8d-bbdd-8b79b829b42b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77412025-04-30T06:42:40.229ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
77422025-04-30T06:42:40.233ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a20d024-b996-4b96-9e35-db5107100420", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77432025-04-30T06:42:40.233ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
77442025-04-30T06:42:40.237ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "859fb2c8-6aff-4a48-b3fc-64464c39db20", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77452025-04-30T06:42:40.237ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
77462025-04-30T06:42:40.240ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b284ad4b-53c2-4486-9799-7d09ec620006", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77472025-04-30T06:42:40.240ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
77482025-04-30T06:42:40.243ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1b73104-fdec-4d90-b5b9-0ab2bd121fdc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77492025-04-30T06:42:40.243ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
77502025-04-30T06:42:40.246ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "026e0e2f-3466-48c7-b7bb-aff10e850fd9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77512025-04-30T06:42:40.247ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
77522025-04-30T06:42:40.248ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "695b5088-5301-4916-a986-3b0b44f03379", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77532025-04-30T06:42:40.248ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
77542025-04-30T06:42:40.250ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c8fea30-06bf-45e0-8974-07ab9be6a3b1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77552025-04-30T06:42:40.250ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
77562025-04-30T06:42:40.254ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd32978f-2ccd-4058-853a-e8aa4ae69d18", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77572025-04-30T06:42:40.254ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
77582025-04-30T06:42:40.254ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
77592025-04-30T06:42:40.256ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "930d2812-8925-4d76-be67-7404c1b18e54", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77602025-04-30T06:42:40.258ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77612025-04-30T06:42:40.259ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77622025-04-30T06:42:40.259ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77632025-04-30T06:42:40.261ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77642025-04-30T06:42:40.262ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77652025-04-30T06:42:40.266ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77662025-04-30T06:42:40.276ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77672025-04-30T06:42:40.276ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77682025-04-30T06:42:40.276ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77692025-04-30T06:42:40.276ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77702025-04-30T06:42:40.276ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77712025-04-30T06:42:40.276ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77722025-04-30T06:42:40.290ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77732025-04-30T06:42:40.293ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77742025-04-30T06:42:40.293ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77752025-04-30T06:42:40.293ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77762025-04-30T06:42:40.295ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77772025-04-30T06:42:40.368ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
77782025-04-30T06:42:40.368ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
77792025-04-30T06:42:40.368ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77802025-04-30T06:42:40.368ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77812025-04-30T06:42:40.368ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77822025-04-30T06:42:40.368ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
77832025-04-30T06:42:40.369ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77842025-04-30T06:42:40.369ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:50778 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
77852025-04-30T06:42:40.370ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:50778 req_id = 4ee77024-09de-45f5-93e3-9e611a32275e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77862025-04-30T06:42:40.374ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 3363 local_addr = [::1]:62607 method = GET remote_addr = [::1]:50778 req_id = 4ee77024-09de-45f5-93e3-9e611a32275e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77872025-04-30T06:42:40.374ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ee77024-09de-45f5-93e3-9e611a32275e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
77882025-04-30T06:42:40.376ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77892025-04-30T06:42:40.376ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77902025-04-30T06:42:40.376ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
77912025-04-30T06:42:40.378ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77922025-04-30T06:42:40.378ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77932025-04-30T06:42:40.378ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77942025-04-30T06:42:40.378ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77952025-04-30T06:42:40.379ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77962025-04-30T06:42:40.379ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77972025-04-30T06:42:40.386ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77982025-04-30T06:42:40.386ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77992025-04-30T06:42:40.386ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
78002025-04-30T06:42:40.387ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78012025-04-30T06:42:40.389ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78022025-04-30T06:42:40.396ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
78032025-04-30T06:42:40.396ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78042025-04-30T06:42:40.396ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78052025-04-30T06:42:40.396ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78062025-04-30T06:42:40.400ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78072025-04-30T06:42:40.401ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
78082025-04-30T06:42:40.402ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78092025-04-30T06:42:40.449ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
78102025-04-30T06:42:40.449ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
78112025-04-30T06:42:40.449ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 466.686693ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
78122025-04-30T06:42:40.449ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
78132025-04-30T06:42:40.449ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
78142025-04-30T06:42:40.450ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78152025-04-30T06:42:40.450ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78162025-04-30T06:42:40.450ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78172025-04-30T06:42:40.450ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78182025-04-30T06:42:40.452ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78192025-04-30T06:42:40.456ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
78202025-04-30T06:42:40.456ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78212025-04-30T06:42:40.456ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78222025-04-30T06:42:40.456ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78232025-04-30T06:42:40.459ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
78242025-04-30T06:42:40.459ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
78252025-04-30T06:42:40.460ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e8 8c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52186 req_id = 6cbb6487-17f5-4424-a1f0-d2eb6db08ebf
78262025-04-30T06:42:40.460ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59532,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52186 req_id = 6cbb6487-17f5-4424-a1f0-d2eb6db08ebf
78272025-04-30T06:42:40.461ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
78282025-04-30T06:42:40.461ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
78292025-04-30T06:42:40.461ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52186 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6cbb6487-17f5-4424-a1f0-d2eb6db08ebf
78302025-04-30T06:42:40.462ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
78312025-04-30T06:42:40.462ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4225, tv_nsec: 261813273 } })
78322025-04-30T06:42:40.462ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
78332025-04-30T06:42:40.463ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fb f1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51501 req_id = a6e8390a-a984-42ff-97ce-0dc8dd4328a2
78342025-04-30T06:42:40.464ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64497,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51501 req_id = a6e8390a-a984-42ff-97ce-0dc8dd4328a2
78352025-04-30T06:42:40.464ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
78362025-04-30T06:42:40.464ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51501 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a6e8390a-a984-42ff-97ce-0dc8dd4328a2
78372025-04-30T06:42:40.552ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
78382025-04-30T06:42:40.553ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78392025-04-30T06:42:40.553ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78402025-04-30T06:42:40.553ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78412025-04-30T06:42:40.553ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
78422025-04-30T06:42:40.554ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78432025-04-30T06:42:40.554ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78442025-04-30T06:42:40.554ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78452025-04-30T06:42:40.555ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 19 be 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64100 req_id = 0676bea8-7d4c-4fb5-a1e1-a20860e3535c
78462025-04-30T06:42:40.555ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6590,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64100 req_id = 0676bea8-7d4c-4fb5-a1e1-a20860e3535c
78472025-04-30T06:42:40.556ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
78482025-04-30T06:42:40.556ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6590, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64100 req_id = 0676bea8-7d4c-4fb5-a1e1-a20860e3535c
78492025-04-30T06:42:40.558ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
78502025-04-30T06:42:40.558ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
78512025-04-30T06:42:40.558ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
78522025-04-30T06:42:40.558ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78532025-04-30T06:42:40.559ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78542025-04-30T06:42:40.559ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78552025-04-30T06:42:40.559ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78562025-04-30T06:42:40.560ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78572025-04-30T06:42:40.591ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78582025-04-30T06:42:40.591ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78592025-04-30T06:42:40.591ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
78602025-04-30T06:42:40.592ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78612025-04-30T06:42:40.596ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3c9cc1c-e101-4935-881b-d640ad98b331", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78622025-04-30T06:42:40.597ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
78632025-04-30T06:42:40.597ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78642025-04-30T06:42:40.597ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
78652025-04-30T06:42:40.598ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78662025-04-30T06:42:40.599ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b922cee2-b556-4c07-a676-d4799e7d87e1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78672025-04-30T06:42:40.599ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
78682025-04-30T06:42:40.602ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c33ac155-4f90-4414-b6b9-09f49a7f12f2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78692025-04-30T06:42:40.602ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
78702025-04-30T06:42:40.604ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ad08d75-7b99-4645-ad20-7c12764e9c88", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78712025-04-30T06:42:40.604ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
78722025-04-30T06:42:40.607ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06317f4c-c1cb-4032-af41-27afe4296979", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78732025-04-30T06:42:40.607ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
78742025-04-30T06:42:40.609ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e09456b-c8cb-4108-bcf9-f7c0d71bfda4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78752025-04-30T06:42:40.609ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
78762025-04-30T06:42:40.612ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7318e620-b7ec-4891-8dda-02caafd44b43", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78772025-04-30T06:42:40.612ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
78782025-04-30T06:42:40.615ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e43a517d-f245-4dc9-a0c1-8eeef65b5f48", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78792025-04-30T06:42:40.615ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
78802025-04-30T06:42:40.616ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2280193f-a43b-402e-84ce-5069bf863233", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78812025-04-30T06:42:40.616ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
78822025-04-30T06:42:40.618ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42f26495-19ba-4c13-b1e2-ce8a8a6c833e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78832025-04-30T06:42:40.618ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
78842025-04-30T06:42:40.619ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6af7df30-4394-47c5-a482-48382b0d5fe8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78852025-04-30T06:42:40.619ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
78862025-04-30T06:42:40.621ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a721d52-a7ef-4241-a9a7-7529f2f40789", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78872025-04-30T06:42:40.621ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
78882025-04-30T06:42:40.625ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "730d7cfe-3e43-4750-be79-c35318e1114e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78892025-04-30T06:42:40.625ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
78902025-04-30T06:42:40.629ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
78912025-04-30T06:42:40.631ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29bbb68b-98a0-4e4c-b9bc-5933edf6662a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78922025-04-30T06:42:40.631ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
78932025-04-30T06:42:40.635ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b7913ae-ca39-4970-b51a-85182b413dac", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78942025-04-30T06:42:40.635ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
78952025-04-30T06:42:40.639ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91d65685-cc4a-4234-80a6-52af30a02741", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78962025-04-30T06:42:40.639ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
78972025-04-30T06:42:40.644ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b05ae77-f6a2-4409-8b96-60d658a2f125", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
78982025-04-30T06:42:40.644ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
78992025-04-30T06:42:40.652ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11a5e005-755f-4087-8c9e-b38a55249dbc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79002025-04-30T06:42:40.652ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
79012025-04-30T06:42:40.656ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84e2c70d-c386-4842-8ad4-8495766ab844", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79022025-04-30T06:42:40.656ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
79032025-04-30T06:42:40.658ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cf9de96-5772-4b5d-9db7-17adf28a4b75", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79042025-04-30T06:42:40.658ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
79052025-04-30T06:42:40.661ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8801c1d-5afa-4da9-84a6-eb5e3fb67140", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79062025-04-30T06:42:40.661ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
79072025-04-30T06:42:40.663ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6484923b-acb6-416c-9722-48db06ed9b01", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79082025-04-30T06:42:40.663ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
79092025-04-30T06:42:40.665ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7316a341-b2e7-49b9-b587-98a9eccfd6b5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79102025-04-30T06:42:40.666ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
79112025-04-30T06:42:40.669ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2510efa1-f771-4697-b039-19afd1777488", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79122025-04-30T06:42:40.669ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
79132025-04-30T06:42:40.674ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76b7e724-0cfc-4c7f-b3ca-9cfaa95d7003", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79142025-04-30T06:42:40.675ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
79152025-04-30T06:42:40.681ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79162025-04-30T06:42:40.681ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79172025-04-30T06:42:40.681ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79182025-04-30T06:42:40.682ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
79192025-04-30T06:42:40.683ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79202025-04-30T06:42:40.684ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d085d923-96d0-4de7-8596-cc9a5569e675", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79212025-04-30T06:42:40.684ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
79222025-04-30T06:42:40.688ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79232025-04-30T06:42:40.696ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79242025-04-30T06:42:40.696ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79252025-04-30T06:42:40.696ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
79262025-04-30T06:42:40.697ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79272025-04-30T06:42:40.702ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79282025-04-30T06:42:40.702ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
79292025-04-30T06:42:40.704ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79302025-04-30T06:42:40.709ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "927681b1-a8ac-4a7b-9367-3841732f7b1e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79312025-04-30T06:42:40.709ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
79322025-04-30T06:42:40.716ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99ab357-b1be-4cb9-b0bc-2d8f682e6da6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79332025-04-30T06:42:40.717ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
79342025-04-30T06:42:40.724ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e27567e-7641-4443-874f-aff816b3575f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79352025-04-30T06:42:40.725ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
79362025-04-30T06:42:40.733ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e889af6-3168-4294-92d3-4be9163772e9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79372025-04-30T06:42:40.733ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
79382025-04-30T06:42:40.735ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "209f4ff5-036c-4bc4-90e4-23a80148422f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79392025-04-30T06:42:40.735ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
79402025-04-30T06:42:40.739ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea4fc038-3575-4a8e-a9d9-ab3b36ecd571", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79412025-04-30T06:42:40.739ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
79422025-04-30T06:42:40.740ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
79432025-04-30T06:42:40.741ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dea11eb-0cc8-4629-8741-0d181737fa77", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79442025-04-30T06:42:40.741ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
79452025-04-30T06:42:40.743ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a985d8a6-09d7-4bae-9a39-c72c6f668e47", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79462025-04-30T06:42:40.743ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
79472025-04-30T06:42:40.746ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc5267bf-aad0-47f4-9528-86ed05c7ed89", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79482025-04-30T06:42:40.747ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
79492025-04-30T06:42:40.751ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f02792e0-eb5d-4981-857b-b71862758815", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79502025-04-30T06:42:40.751ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
79512025-04-30T06:42:40.756ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeab8e64-5693-48fc-aea2-a55419d8ccad", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79522025-04-30T06:42:40.756ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
79532025-04-30T06:42:40.760ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "167da818-c1e2-4da2-b1fb-c8560ee602ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79542025-04-30T06:42:40.760ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
79552025-04-30T06:42:40.769ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ebd1eda-2dc7-4053-b7a0-b326a170a99a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79562025-04-30T06:42:40.770ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
79572025-04-30T06:42:40.773ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fab18466-84d2-4c9e-bfcf-b6941340c0d8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79582025-04-30T06:42:40.773ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
79592025-04-30T06:42:40.776ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ee7fa98-37f8-45fc-87fe-0039d0bfec3c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79602025-04-30T06:42:40.776ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
79612025-04-30T06:42:40.777ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83f7c7c2-7c6c-4b56-9b71-35a04c2ea4ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79622025-04-30T06:42:40.777ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
79632025-04-30T06:42:40.778ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f16a4be8-0efc-483f-be82-8809b9ddf32b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79642025-04-30T06:42:40.778ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
79652025-04-30T06:42:40.779ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e32d782-4026-4e4e-a373-0730543ff12a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79662025-04-30T06:42:40.780ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
79672025-04-30T06:42:40.781ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab5d050b-2da4-4431-bfe1-b6a85fcf999e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:39 GMT"} })
79682025-04-30T06:42:40.781ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79692025-04-30T06:42:40.781ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79702025-04-30T06:42:40.781ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79712025-04-30T06:42:40.781ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79722025-04-30T06:42:40.785ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79732025-04-30T06:42:40.798ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79742025-04-30T06:42:40.799ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79752025-04-30T06:42:40.799ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79762025-04-30T06:42:40.799ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
79772025-04-30T06:42:40.800ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79782025-04-30T06:42:40.800ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79792025-04-30T06:42:40.804ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79802025-04-30T06:42:40.804ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79812025-04-30T06:42:40.804ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79822025-04-30T06:42:40.804ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79832025-04-30T06:42:40.804ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79842025-04-30T06:42:40.805ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79852025-04-30T06:42:40.807ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79862025-04-30T06:42:40.814ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79872025-04-30T06:42:40.814ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79882025-04-30T06:42:40.814ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
79892025-04-30T06:42:40.815ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79902025-04-30T06:42:40.816ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79912025-04-30T06:42:40.820ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79922025-04-30T06:42:40.820ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79932025-04-30T06:42:40.820ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79942025-04-30T06:42:40.821ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79952025-04-30T06:42:40.835ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79962025-04-30T06:42:40.836ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
79972025-04-30T06:42:40.836ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79982025-04-30T06:42:40.917ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
79992025-04-30T06:42:40.917ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
80002025-04-30T06:42:40.918ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:60309 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
80012025-04-30T06:42:40.918ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
80022025-04-30T06:42:40.918ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
80032025-04-30T06:42:40.918ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
80042025-04-30T06:42:40.919ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
80052025-04-30T06:42:40.925ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:60309 req_id = a73c3f55-c424-4cdd-88d4-43a58a570615 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
80062025-04-30T06:42:40.926ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 457 local_addr = [::1]:62607 method = GET remote_addr = [::1]:60309 req_id = a73c3f55-c424-4cdd-88d4-43a58a570615 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
80072025-04-30T06:42:40.928ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
80082025-04-30T06:42:40.928ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
80092025-04-30T06:42:40.929ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a73c3f55-c424-4cdd-88d4-43a58a570615", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80102025-04-30T06:42:40.931ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
80112025-04-30T06:42:40.931ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
80122025-04-30T06:42:40.931ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
80132025-04-30T06:42:40.933ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
80142025-04-30T06:42:40.933ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80152025-04-30T06:42:40.933ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80162025-04-30T06:42:40.933ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80172025-04-30T06:42:40.934ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80182025-04-30T06:42:40.937ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
80192025-04-30T06:42:40.937ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
80202025-04-30T06:42:40.938ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80212025-04-30T06:42:40.943ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
80222025-04-30T06:42:40.943ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80232025-04-30T06:42:40.943ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80242025-04-30T06:42:40.944ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80252025-04-30T06:42:41.002ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
80262025-04-30T06:42:41.011ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
80272025-04-30T06:42:41.012ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
80282025-04-30T06:42:41.012ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 562.48777ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
80292025-04-30T06:42:41.012ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
80302025-04-30T06:42:41.012ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
80312025-04-30T06:42:41.012ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80322025-04-30T06:42:41.012ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80332025-04-30T06:42:41.012ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80342025-04-30T06:42:41.013ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80352025-04-30T06:42:41.015ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80362025-04-30T06:42:41.026ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
80372025-04-30T06:42:41.026ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80382025-04-30T06:42:41.026ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80392025-04-30T06:42:41.026ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80402025-04-30T06:42:41.027ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
80412025-04-30T06:42:41.031ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
80422025-04-30T06:42:41.031ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
80432025-04-30T06:42:41.031ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cc d0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55979 req_id = 818761ee-bfe4-472f-9b80-89006b0b8fe2
80442025-04-30T06:42:41.031ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52432,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55979 req_id = 818761ee-bfe4-472f-9b80-89006b0b8fe2
80452025-04-30T06:42:41.032ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
80462025-04-30T06:42:41.032ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80472025-04-30T06:42:41.033ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55979 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 818761ee-bfe4-472f-9b80-89006b0b8fe2
80482025-04-30T06:42:41.041ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4225, tv_nsec: 839726768 } })
80492025-04-30T06:42:41.041ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
80502025-04-30T06:42:41.042ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 74 dc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e t...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53579 req_id = c62c86ed-9c59-41fd-81f7-4dc3a4f836b2
80512025-04-30T06:42:41.042ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29916,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53579 req_id = c62c86ed-9c59-41fd-81f7-4dc3a4f836b2
80522025-04-30T06:42:41.042ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80532025-04-30T06:42:41.042ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53579 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c62c86ed-9c59-41fd-81f7-4dc3a4f836b2
80542025-04-30T06:42:41.119ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
80552025-04-30T06:42:41.119ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80562025-04-30T06:42:41.120ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80572025-04-30T06:42:41.120ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80582025-04-30T06:42:41.120ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
80592025-04-30T06:42:41.122ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80602025-04-30T06:42:41.125ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 96 01 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52724 req_id = cf5edf32-740e-4dc5-9517-13de8b43918a
80612025-04-30T06:42:41.125ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38401,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52724 req_id = cf5edf32-740e-4dc5-9517-13de8b43918a
80622025-04-30T06:42:41.125ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
80632025-04-30T06:42:41.126ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38401, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52724 req_id = cf5edf32-740e-4dc5-9517-13de8b43918a
80642025-04-30T06:42:41.126ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
80652025-04-30T06:42:41.126ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
80662025-04-30T06:42:41.126ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
80672025-04-30T06:42:41.130ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80682025-04-30T06:42:41.142ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80692025-04-30T06:42:41.142ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80702025-04-30T06:42:41.142ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
80712025-04-30T06:42:41.144ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80722025-04-30T06:42:41.154ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80732025-04-30T06:42:41.155ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
80742025-04-30T06:42:41.155ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80752025-04-30T06:42:41.162ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "668ff452-d0bb-418b-8251-7efca8a63f42", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80762025-04-30T06:42:41.163ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
80772025-04-30T06:42:41.175ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "279093ad-c4d0-4b31-9718-0b797bcde690", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80782025-04-30T06:42:41.175ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
80792025-04-30T06:42:41.184ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad4939d0-87a3-4df8-b38e-f9771af574e7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80802025-04-30T06:42:41.184ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
80812025-04-30T06:42:41.188ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be4b31fe-adde-4594-a9a4-26c5041ee5e3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80822025-04-30T06:42:41.188ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
80832025-04-30T06:42:41.190ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f27b37c0-5a4f-4561-bc1c-c03e9e9b8e35", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80842025-04-30T06:42:41.191ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
80852025-04-30T06:42:41.195ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68a0d315-b668-42cd-a49a-f1daa7022974", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80862025-04-30T06:42:41.195ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
80872025-04-30T06:42:41.203ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
80882025-04-30T06:42:41.211ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05622897-44b5-4ff7-8b9b-c4b6564f2297", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80892025-04-30T06:42:41.211ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
80902025-04-30T06:42:41.244ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7fdc996-0a32-4d77-89d3-0eeab7641d56", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80912025-04-30T06:42:41.244ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
80922025-04-30T06:42:41.245ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8bfa7c2-805a-414c-acd8-2a604c9e171a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80932025-04-30T06:42:41.245ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
80942025-04-30T06:42:41.246ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d02103de-9662-4e3b-8215-b6e47382b246", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80952025-04-30T06:42:41.246ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
80962025-04-30T06:42:41.247ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42364009-7806-4747-aa06-0c7f22874b24", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80972025-04-30T06:42:41.247ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
80982025-04-30T06:42:41.248ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2efa4a00-1ec5-4371-90b5-407ca4ff0514", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
80992025-04-30T06:42:41.249ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
81002025-04-30T06:42:41.250ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "191a155a-c3b8-4a63-99cc-279922a6acfa", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81012025-04-30T06:42:41.251ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
81022025-04-30T06:42:41.258ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81032025-04-30T06:42:41.258ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81042025-04-30T06:42:41.258ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81052025-04-30T06:42:41.258ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
81062025-04-30T06:42:41.259ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81072025-04-30T06:42:41.262ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f74baaa8-cbc0-4b77-b9f7-d7ccb08cbbf2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81082025-04-30T06:42:41.262ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
81092025-04-30T06:42:41.262ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81102025-04-30T06:42:41.270ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81112025-04-30T06:42:41.270ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81122025-04-30T06:42:41.271ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
81132025-04-30T06:42:41.271ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81142025-04-30T06:42:41.274ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81152025-04-30T06:42:41.274ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
81162025-04-30T06:42:41.275ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81172025-04-30T06:42:41.278ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d37198fd-35ce-41e9-b481-706b07d225af", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81182025-04-30T06:42:41.278ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
81192025-04-30T06:42:41.282ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a173c96-b7c9-40dd-b4d2-331c42bc0df4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81202025-04-30T06:42:41.282ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
81212025-04-30T06:42:41.286ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a6f2078-723f-4b62-b8e5-c841f847fff8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81222025-04-30T06:42:41.286ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
81232025-04-30T06:42:41.289ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95cc0cbf-0cb6-407b-85e9-27b7bb6433c6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81242025-04-30T06:42:41.290ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
81252025-04-30T06:42:41.293ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cddb0b5-5d90-4409-8b18-7e2093b224be", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81262025-04-30T06:42:41.293ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
81272025-04-30T06:42:41.294ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72e5ed36-ff43-4f06-9d53-1f074a092a2d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81282025-04-30T06:42:41.295ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
81292025-04-30T06:42:41.296ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d7a28a2-39bd-41bb-bb68-92638959e43b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81302025-04-30T06:42:41.296ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
81312025-04-30T06:42:41.297ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32fcdd77-cec4-4f54-b242-ff6194de0a88", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81322025-04-30T06:42:41.298ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
81332025-04-30T06:42:41.301ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f2de3c8-a15c-4e53-b68b-17d4e213d4a0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81342025-04-30T06:42:41.302ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
81352025-04-30T06:42:41.309ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1c3503e-2f7f-47b9-82a6-62c9b833d0e7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81362025-04-30T06:42:41.309ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
81372025-04-30T06:42:41.310ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
81382025-04-30T06:42:41.313ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab8d2f86-c593-44b5-a873-23b8a20484d4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81392025-04-30T06:42:41.313ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
81402025-04-30T06:42:41.317ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b928b721-e7ff-478f-80e5-94d6cbbeb977", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81412025-04-30T06:42:41.317ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
81422025-04-30T06:42:41.321ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1cfbf59-1f9c-48ec-acbb-c01343760dac", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81432025-04-30T06:42:41.321ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
81442025-04-30T06:42:41.325ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b26151f8-cbb3-4f1f-bfc0-6d19771af58f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81452025-04-30T06:42:41.325ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
81462025-04-30T06:42:41.328ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "adc2e63b-e953-4664-a739-e620900fd50b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81472025-04-30T06:42:41.328ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
81482025-04-30T06:42:41.332ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f026a8a6-ee2a-4c27-ab27-d5c4d443ca77", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81492025-04-30T06:42:41.332ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
81502025-04-30T06:42:41.335ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9b7892a-0feb-4bef-be44-3f124853e59f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81512025-04-30T06:42:41.335ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
81522025-04-30T06:42:41.338ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "192c632e-9767-4b38-8dab-1d4d437a8286", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81532025-04-30T06:42:41.338ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
81542025-04-30T06:42:41.349ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fca8ac28-de70-4353-a276-c823ff77dd51", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81552025-04-30T06:42:41.349ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
81562025-04-30T06:42:41.349ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf56ef31-3d9b-4f7b-ba5a-e45dcbf33d31", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81572025-04-30T06:42:41.349ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
81582025-04-30T06:42:41.351ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2763b19f-4900-43e0-8ded-ac39266b7e30", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81592025-04-30T06:42:41.351ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
81602025-04-30T06:42:41.353ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49d4f3a3-ddda-4b62-8ac6-8599f51a1cba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81612025-04-30T06:42:41.353ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
81622025-04-30T06:42:41.355ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9524051-003f-4073-bc28-2a59d4962f74", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81632025-04-30T06:42:41.355ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
81642025-04-30T06:42:41.357ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ef29d59-4606-4655-b012-3f03f9fd0424", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81652025-04-30T06:42:41.357ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
81662025-04-30T06:42:41.359ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1657158f-5d38-4b3e-bc24-bf491d1868ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81672025-04-30T06:42:41.359ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
81682025-04-30T06:42:41.361ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81692025-04-30T06:42:41.361ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81702025-04-30T06:42:41.361ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81712025-04-30T06:42:41.361ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
81722025-04-30T06:42:41.362ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81732025-04-30T06:42:41.363ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd7b707f-4774-4e10-a543-f50674d582c9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81742025-04-30T06:42:41.364ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
81752025-04-30T06:42:41.367ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81762025-04-30T06:42:41.372ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81772025-04-30T06:42:41.372ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81782025-04-30T06:42:41.373ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
81792025-04-30T06:42:41.373ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81802025-04-30T06:42:41.376ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81812025-04-30T06:42:41.376ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
81822025-04-30T06:42:41.376ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81832025-04-30T06:42:41.377ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84db4651-6376-4f67-9a0f-1921ac91a052", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81842025-04-30T06:42:41.377ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
81852025-04-30T06:42:41.378ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35707d00-93b2-4a66-a5d1-cbf3d26c6926", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81862025-04-30T06:42:41.378ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
81872025-04-30T06:42:41.379ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ce524dc-0bd6-4beb-bf5b-c3879c650a20", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81882025-04-30T06:42:41.379ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
81892025-04-30T06:42:41.381ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88e272b5-397c-417c-b24d-01a1d1668936", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81902025-04-30T06:42:41.381ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
81912025-04-30T06:42:41.382ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc966225-8346-4a72-9f8f-f2669408b320", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
81922025-04-30T06:42:41.382ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81932025-04-30T06:42:41.382ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81942025-04-30T06:42:41.382ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81952025-04-30T06:42:41.382ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81962025-04-30T06:42:41.383ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81972025-04-30T06:42:41.387ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81982025-04-30T06:42:41.392ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81992025-04-30T06:42:41.392ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
82002025-04-30T06:42:41.392ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82012025-04-30T06:42:41.392ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82022025-04-30T06:42:41.392ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82032025-04-30T06:42:41.392ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82042025-04-30T06:42:41.396ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82052025-04-30T06:42:41.401ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
82062025-04-30T06:42:41.401ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82072025-04-30T06:42:41.402ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82082025-04-30T06:42:41.402ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82092025-04-30T06:42:41.462ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
82102025-04-30T06:42:41.462ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
82112025-04-30T06:42:41.463ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:50652 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
82122025-04-30T06:42:41.465ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:50652 req_id = 4af2ee84-7052-4d80-ba52-e846ec32d218 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
82132025-04-30T06:42:41.466ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 483 local_addr = [::1]:62607 method = GET remote_addr = [::1]:50652 req_id = 4af2ee84-7052-4d80-ba52-e846ec32d218 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
82142025-04-30T06:42:41.466ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4af2ee84-7052-4d80-ba52-e846ec32d218", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
82152025-04-30T06:42:41.470ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
82162025-04-30T06:42:41.470ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
82172025-04-30T06:42:41.470ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
82182025-04-30T06:42:41.472ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
82192025-04-30T06:42:41.472ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82202025-04-30T06:42:41.472ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82212025-04-30T06:42:41.472ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82222025-04-30T06:42:41.472ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82232025-04-30T06:42:41.477ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82242025-04-30T06:42:41.482ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82252025-04-30T06:42:41.483ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82262025-04-30T06:42:41.483ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82272025-04-30T06:42:41.483ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82282025-04-30T06:42:41.489ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
82292025-04-30T06:42:41.489ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82302025-04-30T06:42:41.491ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
82312025-04-30T06:42:41.491ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 23c64a50-b8e2-4ee3-8cc7-be861b620eef uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82322025-04-30T06:42:41.491ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 23c64a50-b8e2-4ee3-8cc7-be861b620eef resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82332025-04-30T06:42:41.491ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 23c64a50-b8e2-4ee3-8cc7-be861b620eef roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82342025-04-30T06:42:41.492ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 23c64a50-b8e2-4ee3-8cc7-be861b620eef resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82352025-04-30T06:42:41.496ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 5614 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 23c64a50-b8e2-4ee3-8cc7-be861b620eef response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82362025-04-30T06:42:41.497ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "23c64a50-b8e2-4ee3-8cc7-be861b620eef", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
82372025-04-30T06:42:41.498ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
82382025-04-30T06:42:41.499ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1a046994-b11e-4966-9ca8-d1546cbfe751 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
82392025-04-30T06:42:41.499ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1a046994-b11e-4966-9ca8-d1546cbfe751 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
82402025-04-30T06:42:41.499ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1a046994-b11e-4966-9ca8-d1546cbfe751 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
82412025-04-30T06:42:41.502ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1a046994-b11e-4966-9ca8-d1546cbfe751 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
82422025-04-30T06:42:41.514ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 15416 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1a046994-b11e-4966-9ca8-d1546cbfe751 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
82432025-04-30T06:42:41.515ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a046994-b11e-4966-9ca8-d1546cbfe751", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
82442025-04-30T06:42:41.515ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82452025-04-30T06:42:41.515ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
82462025-04-30T06:42:41.516ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
82472025-04-30T06:42:41.532ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
82482025-04-30T06:42:41.532ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
82492025-04-30T06:42:41.532ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 520.422098ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
82502025-04-30T06:42:41.533ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
82512025-04-30T06:42:41.533ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
82522025-04-30T06:42:41.533ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82532025-04-30T06:42:41.533ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82542025-04-30T06:42:41.533ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82552025-04-30T06:42:41.534ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82562025-04-30T06:42:41.545ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82572025-04-30T06:42:41.545ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82582025-04-30T06:42:41.545ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82592025-04-30T06:42:41.545ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
82602025-04-30T06:42:41.546ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82612025-04-30T06:42:41.547ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82622025-04-30T06:42:41.561ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82632025-04-30T06:42:41.562ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82642025-04-30T06:42:41.562ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82652025-04-30T06:42:41.562ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82662025-04-30T06:42:41.563ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82672025-04-30T06:42:41.563ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82682025-04-30T06:42:41.565ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82692025-04-30T06:42:41.581ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82702025-04-30T06:42:41.581ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82712025-04-30T06:42:41.581ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
82722025-04-30T06:42:41.582ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82732025-04-30T06:42:41.582ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82742025-04-30T06:42:41.582ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82752025-04-30T06:42:41.582ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82762025-04-30T06:42:41.583ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82772025-04-30T06:42:41.583ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
82782025-04-30T06:42:41.583ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
82792025-04-30T06:42:41.585ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82802025-04-30T06:42:41.585ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
82812025-04-30T06:42:41.585ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82822025-04-30T06:42:41.586ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fd 3f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .?..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60907 req_id = 139bb830-f5b3-47df-a361-cd295036b7ca
82832025-04-30T06:42:41.586ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64831,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60907 req_id = 139bb830-f5b3-47df-a361-cd295036b7ca
82842025-04-30T06:42:41.586ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
82852025-04-30T06:42:41.586ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82862025-04-30T06:42:41.586ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60907 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 139bb830-f5b3-47df-a361-cd295036b7ca
82872025-04-30T06:42:41.587ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4226, tv_nsec: 386383146 } })
82882025-04-30T06:42:41.587ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
82892025-04-30T06:42:41.587ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c8 61 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .a..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64809 req_id = b237bdb2-c37b-4231-aca9-35076b4d166e
82902025-04-30T06:42:41.588ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51297,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64809 req_id = b237bdb2-c37b-4231-aca9-35076b4d166e
82912025-04-30T06:42:41.588ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82922025-04-30T06:42:41.588ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64809 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b237bdb2-c37b-4231-aca9-35076b4d166e
82932025-04-30T06:42:41.666ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
82942025-04-30T06:42:41.667ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b8 8c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58332 req_id = 129b81b4-b8b9-49b8-9297-ae186bb3937d
82952025-04-30T06:42:41.667ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47244,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58332 req_id = 129b81b4-b8b9-49b8-9297-ae186bb3937d
82962025-04-30T06:42:41.667ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
82972025-04-30T06:42:41.668ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47244, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58332 req_id = 129b81b4-b8b9-49b8-9297-ae186bb3937d
82982025-04-30T06:42:41.668ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
82992025-04-30T06:42:41.668ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
83002025-04-30T06:42:41.668ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
83012025-04-30T06:42:41.673ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7385a294-cc09-4e7d-be95-69a2151eeace", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83022025-04-30T06:42:41.673ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
83032025-04-30T06:42:41.675ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c60314b-a245-4637-8f51-cceb82d8b586", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83042025-04-30T06:42:41.675ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
83052025-04-30T06:42:41.679ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c03a4b01-ba00-4da2-9bb8-f31019e411c0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83062025-04-30T06:42:41.679ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
83072025-04-30T06:42:41.682ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ab117ca-9b6d-424f-9b3b-b2b764e36cbd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83082025-04-30T06:42:41.682ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
83092025-04-30T06:42:41.685ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a114965-6b1e-4fa7-b9c7-25cba6dc486f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83102025-04-30T06:42:41.685ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
83112025-04-30T06:42:41.689ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbd8ab84-7775-4a5f-8c8d-2a16e2e66880", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83122025-04-30T06:42:41.689ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
83132025-04-30T06:42:41.696ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
83142025-04-30T06:42:41.698ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b55d2e-8e0f-4a77-b60e-b8b919d9114d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83152025-04-30T06:42:41.698ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
83162025-04-30T06:42:41.702ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed44bef8-56f7-491b-a6c9-cfd1dd3b955c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83172025-04-30T06:42:41.702ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
83182025-04-30T06:42:41.703ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbe52c46-d1bc-46e4-94db-e267ba66cfd8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83192025-04-30T06:42:41.703ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
83202025-04-30T06:42:41.705ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5ae76cd-4e27-4670-a06c-92d4bc70c4c8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83212025-04-30T06:42:41.705ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
83222025-04-30T06:42:41.706ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ba01b10-5e8f-4497-bc5e-c3570e2a1bb2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83232025-04-30T06:42:41.707ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
83242025-04-30T06:42:41.709ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "909478c5-69d3-4d55-b515-6b515348c690", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83252025-04-30T06:42:41.709ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
83262025-04-30T06:42:41.711ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d548bde1-92ab-4b61-aa76-49023e2d5954", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83272025-04-30T06:42:41.711ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
83282025-04-30T06:42:41.714ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1daecb9e-5797-4fb4-9fd8-c8b9b6eeed43", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83292025-04-30T06:42:41.714ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
83302025-04-30T06:42:41.716ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bda99c4-8b47-4e94-9397-5f7832c394b0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83312025-04-30T06:42:41.716ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
83322025-04-30T06:42:41.719ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51ba597e-e986-4632-9c59-869c85ea95cc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83332025-04-30T06:42:41.719ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
83342025-04-30T06:42:41.722ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49c0fbad-832e-4018-ac67-b3ae11f975e7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83352025-04-30T06:42:41.722ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
83362025-04-30T06:42:41.726ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1937d56b-752a-4a0a-8099-c12bcd685dbf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83372025-04-30T06:42:41.727ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
83382025-04-30T06:42:41.729ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eb22a20-7613-48f2-96ac-c3239ca7921f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83392025-04-30T06:42:41.729ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
83402025-04-30T06:42:41.730ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4afeb79-3285-40f8-b38b-369f8996f4cf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83412025-04-30T06:42:41.730ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
83422025-04-30T06:42:41.732ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae94c0fa-ac8e-4e46-8a95-4963548fcaac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83432025-04-30T06:42:41.732ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
83442025-04-30T06:42:41.733ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2994be76-2cfc-4c9d-8f9b-761ace80e656", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83452025-04-30T06:42:41.733ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
83462025-04-30T06:42:41.734ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "675d8804-9a83-4b8e-b70c-2d310220e6ec", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83472025-04-30T06:42:41.734ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
83482025-04-30T06:42:41.735ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aea63cb9-6c93-4ab4-9d97-f9b62c14ce73", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83492025-04-30T06:42:41.735ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
83502025-04-30T06:42:41.737ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46eda152-3fe6-4c1c-9b22-9a9c1e825de8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83512025-04-30T06:42:41.738ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
83522025-04-30T06:42:41.742ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cb5a5ca-feb9-4c90-b28b-125a601f7eab", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83532025-04-30T06:42:41.742ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
83542025-04-30T06:42:41.744ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d2bf93e-7a64-494c-941b-b537a09cddc1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83552025-04-30T06:42:41.744ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
83562025-04-30T06:42:41.747ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83572025-04-30T06:42:41.747ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83582025-04-30T06:42:41.747ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83592025-04-30T06:42:41.747ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
83602025-04-30T06:42:41.748ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83612025-04-30T06:42:41.749ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af0a7c2f-56bb-4cf2-813c-4ef7ea9bfff3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83622025-04-30T06:42:41.749ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
83632025-04-30T06:42:41.754ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83642025-04-30T06:42:41.761ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83652025-04-30T06:42:41.761ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83662025-04-30T06:42:41.761ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
83672025-04-30T06:42:41.762ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83682025-04-30T06:42:41.768ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ca7f3fe-6387-439a-b105-da712c0a8f95", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83692025-04-30T06:42:41.768ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
83702025-04-30T06:42:41.769ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83712025-04-30T06:42:41.769ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
83722025-04-30T06:42:41.771ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83732025-04-30T06:42:41.773ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ae4f3ca-f073-45e0-9803-dcbaefb1b18d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83742025-04-30T06:42:41.774ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
83752025-04-30T06:42:41.775ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7df25b93-d6be-408f-8371-3d18f25ce368", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83762025-04-30T06:42:41.775ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
83772025-04-30T06:42:41.776ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "636f1763-5143-4db8-a22e-b4f621fbcafd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83782025-04-30T06:42:41.777ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
83792025-04-30T06:42:41.778ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a42ad65-db3c-453c-85ca-aaebca2fd885", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83802025-04-30T06:42:41.778ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
83812025-04-30T06:42:41.780ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d7d0384-3d4a-4fc1-bc2a-74b7fceda6b9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83822025-04-30T06:42:41.780ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
83832025-04-30T06:42:41.782ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cd9f49a-565c-4c93-8013-e753caa05823", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83842025-04-30T06:42:41.782ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
83852025-04-30T06:42:41.786ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca54a6b-d81b-4045-a766-fe28443d87cf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83862025-04-30T06:42:41.786ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
83872025-04-30T06:42:41.790ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2c2979d-58aa-476a-913e-9371d2fafdf4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83882025-04-30T06:42:41.790ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
83892025-04-30T06:42:41.794ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd2dfac4-e201-48c6-9e39-2f0b17237b8b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83902025-04-30T06:42:41.794ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
83912025-04-30T06:42:41.797ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "474726f6-df32-4cad-b437-461b3a743fcf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83922025-04-30T06:42:41.798ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
83932025-04-30T06:42:41.802ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
83942025-04-30T06:42:41.803ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5a3bd41-fa1f-44e3-907b-7cc8a85a9fb4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83952025-04-30T06:42:41.803ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
83962025-04-30T06:42:41.806ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b503aa2-b618-4102-a220-c851245cd47d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83972025-04-30T06:42:41.806ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
83982025-04-30T06:42:41.807ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fdd5d6d-d0d4-4705-bc21-49e2a110b7a5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
83992025-04-30T06:42:41.807ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
84002025-04-30T06:42:41.808ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "00507784-90ca-4a62-b6a0-d1f686ae60cc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
84012025-04-30T06:42:41.809ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
84022025-04-30T06:42:41.810ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c8f46ac-6b87-4b77-a67a-295464988c3f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
84032025-04-30T06:42:41.810ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
84042025-04-30T06:42:41.812ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6ae2ed5-2708-47cb-87a5-667bf8c9a908", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
84052025-04-30T06:42:41.812ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
84062025-04-30T06:42:41.812ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84072025-04-30T06:42:41.813ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84082025-04-30T06:42:41.813ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84092025-04-30T06:42:41.814ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84102025-04-30T06:42:41.823ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84112025-04-30T06:42:41.828ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
84122025-04-30T06:42:41.828ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
84132025-04-30T06:42:41.828ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84142025-04-30T06:42:41.828ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84152025-04-30T06:42:41.828ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84162025-04-30T06:42:41.829ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84172025-04-30T06:42:41.836ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84182025-04-30T06:42:41.843ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
84192025-04-30T06:42:41.843ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84202025-04-30T06:42:41.843ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84212025-04-30T06:42:41.843ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84222025-04-30T06:42:41.917ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
84232025-04-30T06:42:41.917ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
84242025-04-30T06:42:41.917ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84252025-04-30T06:42:41.917ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84262025-04-30T06:42:41.917ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84272025-04-30T06:42:41.917ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
84282025-04-30T06:42:41.918ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84292025-04-30T06:42:41.918ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:55630 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
84302025-04-30T06:42:41.918ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
84312025-04-30T06:42:41.918ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
84322025-04-30T06:42:41.919ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
84332025-04-30T06:42:41.923ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
84342025-04-30T06:42:41.923ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:55630 req_id = 6adb18d9-6d6f-4dc8-b325-b012f4d58c89 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
84352025-04-30T06:42:41.923ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 465 local_addr = [::1]:62607 method = GET remote_addr = [::1]:55630 req_id = 6adb18d9-6d6f-4dc8-b325-b012f4d58c89 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
84362025-04-30T06:42:41.924ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6adb18d9-6d6f-4dc8-b325-b012f4d58c89", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:40 GMT"} })
84372025-04-30T06:42:41.926ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
84382025-04-30T06:42:41.926ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
84392025-04-30T06:42:41.926ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
84402025-04-30T06:42:41.928ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84412025-04-30T06:42:41.928ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84422025-04-30T06:42:41.928ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84432025-04-30T06:42:41.928ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84442025-04-30T06:42:41.929ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84452025-04-30T06:42:41.929ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84462025-04-30T06:42:41.935ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84472025-04-30T06:42:41.935ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84482025-04-30T06:42:41.935ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
84492025-04-30T06:42:41.936ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84502025-04-30T06:42:41.936ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
84512025-04-30T06:42:41.938ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
84522025-04-30T06:42:41.940ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84532025-04-30T06:42:41.951ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84542025-04-30T06:42:41.951ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84552025-04-30T06:42:41.951ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84562025-04-30T06:42:41.952ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84572025-04-30T06:42:41.955ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84582025-04-30T06:42:41.963ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
84592025-04-30T06:42:41.963ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84602025-04-30T06:42:41.965ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
84612025-04-30T06:42:41.965ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
84622025-04-30T06:42:42.020ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
84632025-04-30T06:42:42.022ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
84642025-04-30T06:42:42.022ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
84652025-04-30T06:42:42.022ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 489.855645ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
84662025-04-30T06:42:42.022ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
84672025-04-30T06:42:42.023ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84682025-04-30T06:42:42.023ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84692025-04-30T06:42:42.023ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84702025-04-30T06:42:42.023ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84712025-04-30T06:42:42.024ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84722025-04-30T06:42:42.031ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84732025-04-30T06:42:42.040ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84742025-04-30T06:42:42.040ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84752025-04-30T06:42:42.040ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84762025-04-30T06:42:42.041ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84772025-04-30T06:42:42.042ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
84782025-04-30T06:42:42.045ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
84792025-04-30T06:42:42.045ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
84802025-04-30T06:42:42.046ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a7 11 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56636 req_id = 0d09d706-a009-4adb-9f72-c104553911d7
84812025-04-30T06:42:42.046ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42769,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56636 req_id = 0d09d706-a009-4adb-9f72-c104553911d7
84822025-04-30T06:42:42.047ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
84832025-04-30T06:42:42.047ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84842025-04-30T06:42:42.047ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56636 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0d09d706-a009-4adb-9f72-c104553911d7
84852025-04-30T06:42:42.048ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4226, tv_nsec: 847274882 } })
84862025-04-30T06:42:42.048ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
84872025-04-30T06:42:42.049ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 74 86 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e t...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49906 req_id = 459e0b7f-ea12-426b-b468-d0bf2bc9848c
84882025-04-30T06:42:42.050ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29830,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49906 req_id = 459e0b7f-ea12-426b-b468-d0bf2bc9848c
84892025-04-30T06:42:42.050ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84902025-04-30T06:42:42.050ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49906 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 459e0b7f-ea12-426b-b468-d0bf2bc9848c
84912025-04-30T06:42:42.124ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
84922025-04-30T06:42:42.125ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84932025-04-30T06:42:42.125ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84942025-04-30T06:42:42.125ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84952025-04-30T06:42:42.125ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
84962025-04-30T06:42:42.125ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84972025-04-30T06:42:42.129ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 20 ab 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55896 req_id = d45a441b-0a07-46c4-8348-16b0f4f043d3
84982025-04-30T06:42:42.129ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8363,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55896 req_id = d45a441b-0a07-46c4-8348-16b0f4f043d3
84992025-04-30T06:42:42.130ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
85002025-04-30T06:42:42.131ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8363, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55896 req_id = d45a441b-0a07-46c4-8348-16b0f4f043d3
85012025-04-30T06:42:42.132ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
85022025-04-30T06:42:42.132ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
85032025-04-30T06:42:42.132ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
85042025-04-30T06:42:42.135ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85052025-04-30T06:42:42.142ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85062025-04-30T06:42:42.142ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85072025-04-30T06:42:42.142ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
85082025-04-30T06:42:42.142ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85092025-04-30T06:42:42.154ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85102025-04-30T06:42:42.154ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
85112025-04-30T06:42:42.155ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85122025-04-30T06:42:42.169ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d773b554-393a-4ccc-bceb-b366e28a4628", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85132025-04-30T06:42:42.169ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
85142025-04-30T06:42:42.206ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ecd2b9c-db8f-4455-8f86-151d4b266b52", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85152025-04-30T06:42:42.207ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
85162025-04-30T06:42:42.216ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f122b87-2385-4efd-ac16-78973214da94", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85172025-04-30T06:42:42.216ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
85182025-04-30T06:42:42.219ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f841db3-cd66-47fc-968d-ce81b91a9e6a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85192025-04-30T06:42:42.219ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
85202025-04-30T06:42:42.221ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5518904-1901-460b-829f-f4a325b7f6f2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85212025-04-30T06:42:42.222ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
85222025-04-30T06:42:42.226ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d715c979-543d-41cd-8e94-afaa5865f35f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85232025-04-30T06:42:42.226ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
85242025-04-30T06:42:42.230ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
85252025-04-30T06:42:42.232ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "240487ea-ef44-4385-982d-bce5fca04e43", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85262025-04-30T06:42:42.232ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
85272025-04-30T06:42:42.234ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "259c686e-956d-44a3-8a05-7dae2ba54c9f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85282025-04-30T06:42:42.234ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
85292025-04-30T06:42:42.236ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edbea76b-dbd8-4a3d-bb4f-f4c0609014b1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85302025-04-30T06:42:42.236ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
85312025-04-30T06:42:42.237ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "495e6c7c-1a67-4187-9c54-c76182733e7a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85322025-04-30T06:42:42.238ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
85332025-04-30T06:42:42.240ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cc80ecf-862a-4635-93ec-56af9a888541", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85342025-04-30T06:42:42.240ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
85352025-04-30T06:42:42.241ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe018501-cc55-4112-b099-1ab949763e3a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85362025-04-30T06:42:42.241ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
85372025-04-30T06:42:42.244ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77c561d4-db69-4519-b68f-5e15c4aecbba", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85382025-04-30T06:42:42.245ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
85392025-04-30T06:42:42.249ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd343db7-36d0-4cde-ab7a-1f101bc5e3a8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85402025-04-30T06:42:42.249ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
85412025-04-30T06:42:42.252ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2337fcb-d12c-4a12-a2d5-af9476637716", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85422025-04-30T06:42:42.252ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
85432025-04-30T06:42:42.255ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a606235a-5b11-4dbd-af74-64327122caa7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85442025-04-30T06:42:42.255ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
85452025-04-30T06:42:42.258ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49a8ad34-bd92-4a42-ae5e-539e02e5c1d3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85462025-04-30T06:42:42.258ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
85472025-04-30T06:42:42.262ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33a1ebc7-0393-4017-9d53-4c23012118c1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85482025-04-30T06:42:42.262ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
85492025-04-30T06:42:42.265ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd6a1060-3920-44f4-a85f-03ae1bd9ff0c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85502025-04-30T06:42:42.265ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
85512025-04-30T06:42:42.266ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b992ef7-6e21-4d36-85b6-709b20add9b4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85522025-04-30T06:42:42.266ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
85532025-04-30T06:42:42.267ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd44272-ba7d-484e-9d96-f1a497bdb840", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85542025-04-30T06:42:42.267ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
85552025-04-30T06:42:42.268ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "785c8d3a-1fa7-4494-8f11-48dd3f5385ff", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85562025-04-30T06:42:42.268ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
85572025-04-30T06:42:42.269ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4bc19d3-82bd-4564-9b4b-b7958c5b11d0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85582025-04-30T06:42:42.269ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
85592025-04-30T06:42:42.271ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0203f29f-208c-45f4-b148-d25380b98e94", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85602025-04-30T06:42:42.271ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
85612025-04-30T06:42:42.273ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7acd89db-930b-4140-88c7-f942275c56e1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85622025-04-30T06:42:42.273ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
85632025-04-30T06:42:42.275ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7a276de-28e1-41cd-a90f-502f0c233c8d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85642025-04-30T06:42:42.276ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
85652025-04-30T06:42:42.278ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcfbca4d-a491-41b3-b794-0a83f9207ccd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85662025-04-30T06:42:42.278ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
85672025-04-30T06:42:42.281ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e4ca8f5-8157-47e2-9cb3-5772cd2c301e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85682025-04-30T06:42:42.281ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
85692025-04-30T06:42:42.281ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85702025-04-30T06:42:42.281ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85712025-04-30T06:42:42.281ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85722025-04-30T06:42:42.281ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
85732025-04-30T06:42:42.282ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85742025-04-30T06:42:42.285ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2929aaec-e1aa-477f-9a0c-afd817fdedd9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85752025-04-30T06:42:42.285ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
85762025-04-30T06:42:42.285ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85772025-04-30T06:42:42.290ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85782025-04-30T06:42:42.290ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85792025-04-30T06:42:42.290ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
85802025-04-30T06:42:42.291ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85812025-04-30T06:42:42.294ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4779185e-2bc0-4831-bf2f-eb994a44203d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85822025-04-30T06:42:42.295ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
85832025-04-30T06:42:42.295ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85842025-04-30T06:42:42.295ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
85852025-04-30T06:42:42.296ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85862025-04-30T06:42:42.298ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42e394a4-a907-485a-a158-fce077dfc057", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85872025-04-30T06:42:42.299ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
85882025-04-30T06:42:42.301ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "894fc21c-5ff7-4b12-a1a2-be5c23d8ed05", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85892025-04-30T06:42:42.301ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
85902025-04-30T06:42:42.302ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0375d026-3cc5-4c6e-950a-0513ae253fcc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85912025-04-30T06:42:42.302ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
85922025-04-30T06:42:42.304ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19d2ca85-c862-4c33-a06e-c9662b8d7b19", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85932025-04-30T06:42:42.304ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
85942025-04-30T06:42:42.306ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4395e8d-3245-4620-adb7-fc2eecb0c8fe", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85952025-04-30T06:42:42.306ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
85962025-04-30T06:42:42.309ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22998fb3-c5fb-4e14-bde5-276889cb2e33", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85972025-04-30T06:42:42.309ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
85982025-04-30T06:42:42.312ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d1a2508-8502-47d5-86b6-bf43540de1ce", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
85992025-04-30T06:42:42.312ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
86002025-04-30T06:42:42.316ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39241757-4150-42ac-8a78-9b64c45af6ec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
86012025-04-30T06:42:42.316ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
86022025-04-30T06:42:42.320ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c3a318a-2bb3-4b62-90e3-15fe249da0f8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
86032025-04-30T06:42:42.320ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
86042025-04-30T06:42:42.323ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "763380ad-4562-44ea-bd60-5eacb75affb6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
86052025-04-30T06:42:42.323ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
86062025-04-30T06:42:42.327ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "adb869c9-b354-48ba-a38b-fa0b62e7d8cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
86072025-04-30T06:42:42.327ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
86082025-04-30T06:42:42.330ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
86092025-04-30T06:42:42.330ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7960b0d4-06e7-432e-b3a0-f68a8f76fd81", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
86102025-04-30T06:42:42.330ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
86112025-04-30T06:42:42.332ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf5668b8-5612-459c-b7d1-1549936172d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
86122025-04-30T06:42:42.332ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
86132025-04-30T06:42:42.334ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e97e4d79-2670-4693-8e70-807589c4a9ee", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
86142025-04-30T06:42:42.334ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
86152025-04-30T06:42:42.334ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47cc7196-9131-4b85-a974-4e5b17838033", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
86162025-04-30T06:42:42.334ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
86172025-04-30T06:42:42.334ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86182025-04-30T06:42:42.334ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86192025-04-30T06:42:42.334ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86202025-04-30T06:42:42.335ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86212025-04-30T06:42:42.340ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86222025-04-30T06:42:42.345ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
86232025-04-30T06:42:42.345ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
86242025-04-30T06:42:42.345ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86252025-04-30T06:42:42.345ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86262025-04-30T06:42:42.345ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86272025-04-30T06:42:42.345ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86282025-04-30T06:42:42.349ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86292025-04-30T06:42:42.352ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
86302025-04-30T06:42:42.352ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86312025-04-30T06:42:42.352ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86322025-04-30T06:42:42.352ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86332025-04-30T06:42:42.456ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
86342025-04-30T06:42:42.457ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
86352025-04-30T06:42:42.460ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86362025-04-30T06:42:42.460ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86372025-04-30T06:42:42.460ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86382025-04-30T06:42:42.460ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
86392025-04-30T06:42:42.462ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86402025-04-30T06:42:42.462ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:58135 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
86412025-04-30T06:42:42.462ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:58135 req_id = 3f3fde2a-4896-413f-8bc9-422b8fd3a858 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86422025-04-30T06:42:42.463ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 781 local_addr = [::1]:62607 method = GET remote_addr = [::1]:58135 req_id = 3f3fde2a-4896-413f-8bc9-422b8fd3a858 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86432025-04-30T06:42:42.465ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f3fde2a-4896-413f-8bc9-422b8fd3a858", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
86442025-04-30T06:42:42.468ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86452025-04-30T06:42:42.468ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86462025-04-30T06:42:42.468ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
86472025-04-30T06:42:42.472ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86482025-04-30T06:42:42.473ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86492025-04-30T06:42:42.473ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86502025-04-30T06:42:42.473ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86512025-04-30T06:42:42.476ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86522025-04-30T06:42:42.477ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86532025-04-30T06:42:42.488ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86542025-04-30T06:42:42.488ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86552025-04-30T06:42:42.488ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
86562025-04-30T06:42:42.489ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86572025-04-30T06:42:42.492ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86582025-04-30T06:42:42.498ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86592025-04-30T06:42:42.498ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86602025-04-30T06:42:42.498ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86612025-04-30T06:42:42.499ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86622025-04-30T06:42:42.500ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86632025-04-30T06:42:42.500ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
86642025-04-30T06:42:42.501ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86652025-04-30T06:42:42.553ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86662025-04-30T06:42:42.553ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86672025-04-30T06:42:42.555ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
86682025-04-30T06:42:42.555ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
86692025-04-30T06:42:42.556ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 533.662458ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
86702025-04-30T06:42:42.556ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
86712025-04-30T06:42:42.556ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86722025-04-30T06:42:42.557ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86732025-04-30T06:42:42.557ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86742025-04-30T06:42:42.557ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86752025-04-30T06:42:42.559ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86762025-04-30T06:42:42.561ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86772025-04-30T06:42:42.562ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86782025-04-30T06:42:42.562ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86792025-04-30T06:42:42.562ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86802025-04-30T06:42:42.564ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86812025-04-30T06:42:42.576ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86822025-04-30T06:42:42.576ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86832025-04-30T06:42:42.576ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86842025-04-30T06:42:42.576ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86852025-04-30T06:42:42.581ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
86862025-04-30T06:42:42.582ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
86872025-04-30T06:42:42.582ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
86882025-04-30T06:42:42.583ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8d c0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61902 req_id = 432ea893-f583-4d83-960c-3d9115a4734a
86892025-04-30T06:42:42.583ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36288,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61902 req_id = 432ea893-f583-4d83-960c-3d9115a4734a
86902025-04-30T06:42:42.584ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
86912025-04-30T06:42:42.584ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86922025-04-30T06:42:42.584ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61902 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 432ea893-f583-4d83-960c-3d9115a4734a
86932025-04-30T06:42:42.585ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4227, tv_nsec: 384196193 } })
86942025-04-30T06:42:42.585ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
86952025-04-30T06:42:42.585ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bc 32 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .2..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58404 req_id = 0e29634d-829a-45db-bb6f-a43233f13f13
86962025-04-30T06:42:42.585ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48178,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58404 req_id = 0e29634d-829a-45db-bb6f-a43233f13f13
86972025-04-30T06:42:42.586ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86982025-04-30T06:42:42.586ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58404 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0e29634d-829a-45db-bb6f-a43233f13f13
86992025-04-30T06:42:42.660ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
87002025-04-30T06:42:42.660ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87012025-04-30T06:42:42.660ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87022025-04-30T06:42:42.660ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87032025-04-30T06:42:42.660ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
87042025-04-30T06:42:42.661ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87052025-04-30T06:42:42.662ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7f 7e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .~..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54480 req_id = 70763eae-c33d-4c6a-ac5b-7bee71b0ea16
87062025-04-30T06:42:42.662ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32638,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54480 req_id = 70763eae-c33d-4c6a-ac5b-7bee71b0ea16
87072025-04-30T06:42:42.662ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
87082025-04-30T06:42:42.662ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32638, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54480 req_id = 70763eae-c33d-4c6a-ac5b-7bee71b0ea16
87092025-04-30T06:42:42.662ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
87102025-04-30T06:42:42.662ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
87112025-04-30T06:42:42.663ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
87122025-04-30T06:42:42.665ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87132025-04-30T06:42:42.704ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87142025-04-30T06:42:42.704ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87152025-04-30T06:42:42.704ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
87162025-04-30T06:42:42.705ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87172025-04-30T06:42:42.709ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fe6056a-7f0b-4fc1-b320-40b729cb5aaa", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87182025-04-30T06:42:42.710ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
87192025-04-30T06:42:42.710ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87202025-04-30T06:42:42.710ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
87212025-04-30T06:42:42.710ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87222025-04-30T06:42:42.713ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d179308-f49d-4bee-ae9f-a482e0e6c498", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87232025-04-30T06:42:42.715ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
87242025-04-30T06:42:42.725ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3311e3b7-1aab-4163-95bf-f42cd164486c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87252025-04-30T06:42:42.725ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
87262025-04-30T06:42:42.735ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20516dc5-9579-4f26-9547-fcbbb587330d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87272025-04-30T06:42:42.735ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
87282025-04-30T06:42:42.738ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaff3e80-3626-4564-8c65-7bd5a94a7ad0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87292025-04-30T06:42:42.739ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
87302025-04-30T06:42:42.743ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92371575-8f14-43f3-abaa-ee3bc38f4846", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87312025-04-30T06:42:42.744ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
87322025-04-30T06:42:42.747ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99c9cdb2-8a6e-4acc-b6df-7b5f753ab9f1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87332025-04-30T06:42:42.747ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
87342025-04-30T06:42:42.753ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
87352025-04-30T06:42:42.753ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57b7ee23-562d-4b6f-a12c-f758379d9de1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87362025-04-30T06:42:42.753ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
87372025-04-30T06:42:42.755ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40c92c9f-82f9-4acc-87c5-a18862f789dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87382025-04-30T06:42:42.755ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
87392025-04-30T06:42:42.756ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb040a3f-2d05-4874-ab15-6ba2423caa71", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87402025-04-30T06:42:42.756ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
87412025-04-30T06:42:42.757ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff1da52a-d397-4f14-9079-6b739c8ea070", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87422025-04-30T06:42:42.758ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
87432025-04-30T06:42:42.758ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c5412b3-8b03-4e14-8d2e-d7728267def1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87442025-04-30T06:42:42.759ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
87452025-04-30T06:42:42.760ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1532214-b4ab-4746-a582-5b237a526eb1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87462025-04-30T06:42:42.760ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
87472025-04-30T06:42:42.762ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6f86450-f886-485e-8daf-6b3ee867f177", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87482025-04-30T06:42:42.762ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
87492025-04-30T06:42:42.764ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0a1b39b-a45a-4faa-94ae-e80cd0d546c6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87502025-04-30T06:42:42.764ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
87512025-04-30T06:42:42.767ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "791ab74f-391d-41bd-8297-0d3f5175d542", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87522025-04-30T06:42:42.767ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
87532025-04-30T06:42:42.769ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62326841-74a2-48d1-8e4d-aa8b88261349", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87542025-04-30T06:42:42.769ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
87552025-04-30T06:42:42.771ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "636f17fa-b162-4bcd-972e-f7db6e87fbca", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87562025-04-30T06:42:42.771ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
87572025-04-30T06:42:42.773ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d67b0a6e-2127-4da6-847d-905c0ff9abd3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87582025-04-30T06:42:42.773ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
87592025-04-30T06:42:42.774ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bff5f86-d658-4abc-884b-98d5ce4ebe3c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87602025-04-30T06:42:42.774ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
87612025-04-30T06:42:42.775ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27cd7300-7a96-4583-947e-e40fed02e818", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87622025-04-30T06:42:42.775ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
87632025-04-30T06:42:42.776ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cd0f90a-789c-43e9-8d98-c3d32fbaab33", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87642025-04-30T06:42:42.777ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
87652025-04-30T06:42:42.778ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69732320-0255-4eb0-9b16-7e6647a34e8b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87662025-04-30T06:42:42.778ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
87672025-04-30T06:42:42.780ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10fec1b-d78a-40d7-bd33-0c6d2991b336", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87682025-04-30T06:42:42.780ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
87692025-04-30T06:42:42.782ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d329f222-d50f-4567-8601-812035e00a17", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87702025-04-30T06:42:42.782ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
87712025-04-30T06:42:42.784ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "444f673c-2cdf-4641-98a8-38b46b49aaa2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87722025-04-30T06:42:42.785ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
87732025-04-30T06:42:42.787ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef4c2be7-81cf-4e92-8413-72b3e53aae9b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87742025-04-30T06:42:42.787ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
87752025-04-30T06:42:42.789ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee01e187-652d-4e7f-bacb-8cbdeb40b9e5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87762025-04-30T06:42:42.791ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
87772025-04-30T06:42:42.806ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87782025-04-30T06:42:42.806ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87792025-04-30T06:42:42.806ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87802025-04-30T06:42:42.806ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
87812025-04-30T06:42:42.807ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87822025-04-30T06:42:42.810ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "383229f1-956b-4eda-88fe-d3e413e9b15a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87832025-04-30T06:42:42.810ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
87842025-04-30T06:42:42.810ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87852025-04-30T06:42:42.818ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87862025-04-30T06:42:42.819ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87872025-04-30T06:42:42.819ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
87882025-04-30T06:42:42.819ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87892025-04-30T06:42:42.821ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87902025-04-30T06:42:42.821ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
87912025-04-30T06:42:42.822ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87922025-04-30T06:42:42.823ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41da5635-4259-4aa0-bb91-ffb9a2e199a9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87932025-04-30T06:42:42.823ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
87942025-04-30T06:42:42.827ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32093c3a-1c63-48a7-af02-5b6c5da24cec", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87952025-04-30T06:42:42.827ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
87962025-04-30T06:42:42.832ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "abfbf2c3-efc6-4164-9491-fcea6448c512", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87972025-04-30T06:42:42.832ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
87982025-04-30T06:42:42.834ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db340287-6fce-41f3-a67a-2679601bf9e9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
87992025-04-30T06:42:42.834ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
88002025-04-30T06:42:42.838ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "016d012d-dc16-4dc9-8119-7811f4d5e3fc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88012025-04-30T06:42:42.838ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
88022025-04-30T06:42:42.842ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af163634-fcd3-4b02-b0d9-74216b328051", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88032025-04-30T06:42:42.842ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
88042025-04-30T06:42:42.847ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fe82ec7-7b38-4bd5-b235-2ef548a668eb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88052025-04-30T06:42:42.848ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
88062025-04-30T06:42:42.852ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5e13061-fc02-428c-b995-f7dbb01fe8fd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88072025-04-30T06:42:42.852ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
88082025-04-30T06:42:42.857ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
88092025-04-30T06:42:42.860ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5b3a6c9-3ad9-4e5e-a48e-3383e544ccda", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88102025-04-30T06:42:42.860ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
88112025-04-30T06:42:42.862ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a9ea4a5-2bb6-46b2-a6f1-a966c3896866", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88122025-04-30T06:42:42.862ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
88132025-04-30T06:42:42.864ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dee37e66-2051-48c9-a4c4-b2cf6997f419", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88142025-04-30T06:42:42.864ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
88152025-04-30T06:42:42.866ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd803d24-43e8-4892-b067-38a4d2c3b4a8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88162025-04-30T06:42:42.866ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
88172025-04-30T06:42:42.869ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67088d87-25cf-42f1-8177-50fb2f560565", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88182025-04-30T06:42:42.869ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
88192025-04-30T06:42:42.872ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a50d5539-5421-4c91-aca1-505cb89c43ff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88202025-04-30T06:42:42.872ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
88212025-04-30T06:42:42.873ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bed75ec-fc9b-4cc4-a866-36e6ee82502a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88222025-04-30T06:42:42.874ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
88232025-04-30T06:42:42.878ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94698ada-d92c-4cf5-8a33-6c5f861a7bde", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:41 GMT"} })
88242025-04-30T06:42:42.878ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88252025-04-30T06:42:42.878ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88262025-04-30T06:42:42.878ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88272025-04-30T06:42:42.878ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88282025-04-30T06:42:42.880ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88292025-04-30T06:42:42.883ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88302025-04-30T06:42:42.892ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88312025-04-30T06:42:42.892ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88322025-04-30T06:42:42.892ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88332025-04-30T06:42:42.892ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88342025-04-30T06:42:42.892ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88352025-04-30T06:42:42.893ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88362025-04-30T06:42:42.908ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88372025-04-30T06:42:42.908ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88382025-04-30T06:42:42.908ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88392025-04-30T06:42:42.908ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
88402025-04-30T06:42:42.908ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88412025-04-30T06:42:42.908ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
88422025-04-30T06:42:42.909ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
88432025-04-30T06:42:42.909ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
88442025-04-30T06:42:42.910ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
88452025-04-30T06:42:42.916ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88462025-04-30T06:42:42.920ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88472025-04-30T06:42:42.920ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88482025-04-30T06:42:42.920ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88492025-04-30T06:42:42.921ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88502025-04-30T06:42:42.924ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88512025-04-30T06:42:42.929ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88522025-04-30T06:42:42.929ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88532025-04-30T06:42:42.929ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
88542025-04-30T06:42:42.930ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88552025-04-30T06:42:42.930ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
88562025-04-30T06:42:42.930ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
88572025-04-30T06:42:42.992ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
88582025-04-30T06:42:42.992ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
88592025-04-30T06:42:42.993ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:32835 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
88602025-04-30T06:42:42.995ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88612025-04-30T06:42:42.995ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
88622025-04-30T06:42:42.995ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88632025-04-30T06:42:42.996ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:32835 req_id = 584a51ef-4c1a-4df9-88f8-4eb2b667ed81 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88642025-04-30T06:42:42.998ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 2116 local_addr = [::1]:62607 method = GET remote_addr = [::1]:32835 req_id = 584a51ef-4c1a-4df9-88f8-4eb2b667ed81 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88652025-04-30T06:42:42.998ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
88662025-04-30T06:42:42.998ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
88672025-04-30T06:42:43.000ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "584a51ef-4c1a-4df9-88f8-4eb2b667ed81", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
88682025-04-30T06:42:43.007ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88692025-04-30T06:42:43.007ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88702025-04-30T06:42:43.008ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
88712025-04-30T06:42:43.010ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88722025-04-30T06:42:43.010ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88732025-04-30T06:42:43.010ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88742025-04-30T06:42:43.010ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88752025-04-30T06:42:43.011ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88762025-04-30T06:42:43.015ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
88772025-04-30T06:42:43.016ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88782025-04-30T06:42:43.029ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88792025-04-30T06:42:43.029ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88802025-04-30T06:42:43.030ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88812025-04-30T06:42:43.030ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88822025-04-30T06:42:43.084ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
88832025-04-30T06:42:43.092ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
88842025-04-30T06:42:43.094ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
88852025-04-30T06:42:43.094ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 537.66335ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
88862025-04-30T06:42:43.103ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
88872025-04-30T06:42:43.103ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88882025-04-30T06:42:43.103ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88892025-04-30T06:42:43.103ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88902025-04-30T06:42:43.103ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88912025-04-30T06:42:43.104ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88922025-04-30T06:42:43.117ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88932025-04-30T06:42:43.123ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88942025-04-30T06:42:43.124ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88952025-04-30T06:42:43.124ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88962025-04-30T06:42:43.125ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88972025-04-30T06:42:43.129ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
88982025-04-30T06:42:43.129ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
88992025-04-30T06:42:43.136ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 7b eb 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 {............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59463 req_id = c6841a84-9674-41e4-b708-3077521f6d1a
89002025-04-30T06:42:43.136ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31723,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59463 req_id = c6841a84-9674-41e4-b708-3077521f6d1a
89012025-04-30T06:42:43.137ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
89022025-04-30T06:42:43.137ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89032025-04-30T06:42:43.138ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59463 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c6841a84-9674-41e4-b708-3077521f6d1a
89042025-04-30T06:42:43.138ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89052025-04-30T06:42:43.138ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89062025-04-30T06:42:43.138ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89072025-04-30T06:42:43.138ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
89082025-04-30T06:42:43.139ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89092025-04-30T06:42:43.139ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4227, tv_nsec: 938668112 } })
89102025-04-30T06:42:43.139ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
89112025-04-30T06:42:43.142ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 12 36 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .6..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50276 req_id = fc277592-02ca-4fe7-95a1-9b3ab5c57236
89122025-04-30T06:42:43.142ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4662,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50276 req_id = fc277592-02ca-4fe7-95a1-9b3ab5c57236
89132025-04-30T06:42:43.142ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89142025-04-30T06:42:43.142ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50276 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fc277592-02ca-4fe7-95a1-9b3ab5c57236
89152025-04-30T06:42:43.143ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89162025-04-30T06:42:43.149ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89172025-04-30T06:42:43.149ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89182025-04-30T06:42:43.149ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
89192025-04-30T06:42:43.150ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89202025-04-30T06:42:43.225ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
89212025-04-30T06:42:43.231ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 40 d7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c @............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52777 req_id = a4184afb-de65-41be-8862-b3cd54a1f7c6
89222025-04-30T06:42:43.232ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16599,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52777 req_id = a4184afb-de65-41be-8862-b3cd54a1f7c6
89232025-04-30T06:42:43.232ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
89242025-04-30T06:42:43.232ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16599, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52777 req_id = a4184afb-de65-41be-8862-b3cd54a1f7c6
89252025-04-30T06:42:43.233ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89262025-04-30T06:42:43.233ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
89272025-04-30T06:42:43.235ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89282025-04-30T06:42:43.235ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
89292025-04-30T06:42:43.235ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
89302025-04-30T06:42:43.235ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
89312025-04-30T06:42:43.240ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5a247a6-509b-478b-947c-e6bd376e4049", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89322025-04-30T06:42:43.240ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
89332025-04-30T06:42:43.243ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9852dacd-0644-42f3-bed3-35e5f242dce2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89342025-04-30T06:42:43.244ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
89352025-04-30T06:42:43.246ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "772628de-db53-4181-80c0-678d7c46e8b9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89362025-04-30T06:42:43.246ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
89372025-04-30T06:42:43.248ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88f5b3c8-a256-423e-aceb-2b245ee155d1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89382025-04-30T06:42:43.248ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
89392025-04-30T06:42:43.250ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdfc730f-b2b4-4ba1-8a7a-378b02bbdfa2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89402025-04-30T06:42:43.250ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
89412025-04-30T06:42:43.254ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0803dc77-80af-430e-b42e-5843584c1688", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89422025-04-30T06:42:43.254ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
89432025-04-30T06:42:43.256ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "961769de-38c8-4853-a170-41845c64f3fd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89442025-04-30T06:42:43.256ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
89452025-04-30T06:42:43.259ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2db1146b-91df-4288-9578-f53744438ee5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89462025-04-30T06:42:43.259ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
89472025-04-30T06:42:43.260ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f60526c3-fb1c-4a43-a1e0-012a90c305c0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89482025-04-30T06:42:43.260ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
89492025-04-30T06:42:43.264ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21ada60f-9274-4f4f-ba10-cf80b206e9b1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89502025-04-30T06:42:43.265ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
89512025-04-30T06:42:43.270ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "556ac390-0600-43e0-9f1b-e5af2f2193ff", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89522025-04-30T06:42:43.270ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
89532025-04-30T06:42:43.272ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
89542025-04-30T06:42:43.273ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e2dcc4a-0516-4594-b156-bc0474964c24", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89552025-04-30T06:42:43.273ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
89562025-04-30T06:42:43.275ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ec0ebb2-497a-40b7-b1a1-bcbd101ddd71", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89572025-04-30T06:42:43.276ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
89582025-04-30T06:42:43.278ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79a75536-2afe-4a99-b6f9-14b1f38b13fb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89592025-04-30T06:42:43.278ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
89602025-04-30T06:42:43.280ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16b4a5a1-9677-4def-8b14-9885a68cd9af", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89612025-04-30T06:42:43.280ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
89622025-04-30T06:42:43.287ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91bd77ba-a07b-4fd0-a81e-9ca3201ce430", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89632025-04-30T06:42:43.287ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
89642025-04-30T06:42:43.290ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57ffd679-1a48-452d-81bc-529ee0606312", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89652025-04-30T06:42:43.290ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
89662025-04-30T06:42:43.293ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d35d1c3-099f-4fbb-8247-0e36fa3b0b38", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89672025-04-30T06:42:43.293ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
89682025-04-30T06:42:43.298ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6084e043-3b9d-4b86-ae1f-3da416a2e848", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89692025-04-30T06:42:43.298ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
89702025-04-30T06:42:43.299ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f33fbf9e-b309-4f69-bdb0-01f1f5de3481", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89712025-04-30T06:42:43.299ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
89722025-04-30T06:42:43.301ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a38596ce-4eec-4e58-bece-2a1510712e6e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89732025-04-30T06:42:43.301ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
89742025-04-30T06:42:43.303ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c98c3d31-1070-42cd-8442-391269e99db9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89752025-04-30T06:42:43.303ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
89762025-04-30T06:42:43.304ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fbc663c-a807-40a2-8697-1b0158e86d40", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89772025-04-30T06:42:43.304ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
89782025-04-30T06:42:43.306ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6044902b-c6ff-41fe-acdd-690c47f6ea7f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89792025-04-30T06:42:43.306ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
89802025-04-30T06:42:43.308ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "841b019c-797d-4686-84a5-f55eaa50df2e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89812025-04-30T06:42:43.310ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
89822025-04-30T06:42:43.314ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b955c04b-159e-45ea-b992-f2ab8faaf5b4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89832025-04-30T06:42:43.315ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
89842025-04-30T06:42:43.317ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32af39cc-983c-486a-b112-772b816f82ae", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89852025-04-30T06:42:43.317ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
89862025-04-30T06:42:43.321ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70ef8f93-083b-4ba8-b64a-d1bfd466f881", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89872025-04-30T06:42:43.321ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
89882025-04-30T06:42:43.324ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89892025-04-30T06:42:43.324ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89902025-04-30T06:42:43.324ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89912025-04-30T06:42:43.324ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
89922025-04-30T06:42:43.327ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89932025-04-30T06:42:43.332ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89942025-04-30T06:42:43.342ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89952025-04-30T06:42:43.342ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89962025-04-30T06:42:43.342ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
89972025-04-30T06:42:43.343ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89982025-04-30T06:42:43.344ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45678de6-2421-4726-bace-586e5181646f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
89992025-04-30T06:42:43.344ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
90002025-04-30T06:42:43.345ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90012025-04-30T06:42:43.345ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
90022025-04-30T06:42:43.349ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90032025-04-30T06:42:43.359ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f469aa8a-c4c3-447e-b72a-385cd690f81a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90042025-04-30T06:42:43.359ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
90052025-04-30T06:42:43.362ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bc99546-831f-43ef-8547-1808979a4e8d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90062025-04-30T06:42:43.362ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
90072025-04-30T06:42:43.367ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5c5df08-8482-4236-8edc-ce2c39e96e99", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90082025-04-30T06:42:43.368ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
90092025-04-30T06:42:43.368ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0391397b-9d74-4707-8440-77bf3d970f0e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90102025-04-30T06:42:43.369ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
90112025-04-30T06:42:43.370ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e7b0eec-bbe0-4a75-a493-c15d7a5ecc39", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90122025-04-30T06:42:43.370ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
90132025-04-30T06:42:43.372ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a2bdd6c-21fd-400e-aaaa-11e2868353dd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90142025-04-30T06:42:43.372ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
90152025-04-30T06:42:43.378ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "931ce9a3-f4e0-4361-a19f-06f69a8c8f30", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90162025-04-30T06:42:43.378ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
90172025-04-30T06:42:43.382ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27250ec1-cf02-4324-9618-98e67ca1dbdf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90182025-04-30T06:42:43.383ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
90192025-04-30T06:42:43.390ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6966b7be-5c68-483a-9298-3be58f6296e2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90202025-04-30T06:42:43.390ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
90212025-04-30T06:42:43.393ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b502916-89f8-4308-bb47-21b9f1ef08d5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90222025-04-30T06:42:43.393ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
90232025-04-30T06:42:43.400ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
90242025-04-30T06:42:43.401ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56b1a337-07a4-4705-92bd-cd695dbde496", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90252025-04-30T06:42:43.401ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
90262025-04-30T06:42:43.406ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "528ae8f7-cff4-4bc6-9fbe-a3cc43e266ed", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90272025-04-30T06:42:43.406ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
90282025-04-30T06:42:43.407ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b488cc9e-f15d-4b61-871f-9c2ff63c5119", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90292025-04-30T06:42:43.407ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
90302025-04-30T06:42:43.408ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b4e2be7-e217-4ed8-9164-eadd992739e2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90312025-04-30T06:42:43.408ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
90322025-04-30T06:42:43.410ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b597c1f-8359-4c83-b040-db19e013d33d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90332025-04-30T06:42:43.410ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
90342025-04-30T06:42:43.411ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0dbd00c-4455-4f25-9cf0-d41d0d5613a8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90352025-04-30T06:42:43.411ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90362025-04-30T06:42:43.411ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90372025-04-30T06:42:43.411ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90382025-04-30T06:42:43.411ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90392025-04-30T06:42:43.412ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90402025-04-30T06:42:43.414ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90412025-04-30T06:42:43.419ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90422025-04-30T06:42:43.419ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90432025-04-30T06:42:43.419ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90442025-04-30T06:42:43.419ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90452025-04-30T06:42:43.419ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90462025-04-30T06:42:43.421ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90472025-04-30T06:42:43.423ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90482025-04-30T06:42:43.430ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90492025-04-30T06:42:43.430ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90502025-04-30T06:42:43.430ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90512025-04-30T06:42:43.431ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90522025-04-30T06:42:43.501ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
90532025-04-30T06:42:43.501ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
90542025-04-30T06:42:43.502ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90552025-04-30T06:42:43.502ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90562025-04-30T06:42:43.502ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90572025-04-30T06:42:43.502ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
90582025-04-30T06:42:43.503ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90592025-04-30T06:42:43.503ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:61376 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
90602025-04-30T06:42:43.503ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
90612025-04-30T06:42:43.504ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90622025-04-30T06:42:43.504ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:61376 req_id = ef216be4-5ee9-484c-9e33-fbb4eb156296 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90632025-04-30T06:42:43.504ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 533 local_addr = [::1]:62607 method = GET remote_addr = [::1]:61376 req_id = ef216be4-5ee9-484c-9e33-fbb4eb156296 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90642025-04-30T06:42:43.505ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = dfe977d8-2b8e-43a1-9741-5bb40bc63ad0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90652025-04-30T06:42:43.505ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = dfe977d8-2b8e-43a1-9741-5bb40bc63ad0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90662025-04-30T06:42:43.505ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = dfe977d8-2b8e-43a1-9741-5bb40bc63ad0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90672025-04-30T06:42:43.506ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = dfe977d8-2b8e-43a1-9741-5bb40bc63ad0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90682025-04-30T06:42:43.506ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef216be4-5ee9-484c-9e33-fbb4eb156296", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90692025-04-30T06:42:43.508ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90702025-04-30T06:42:43.508ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90712025-04-30T06:42:43.508ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
90722025-04-30T06:42:43.511ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90732025-04-30T06:42:43.511ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90742025-04-30T06:42:43.511ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90752025-04-30T06:42:43.511ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90762025-04-30T06:42:43.512ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90772025-04-30T06:42:43.513ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90782025-04-30T06:42:43.529ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90792025-04-30T06:42:43.529ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90802025-04-30T06:42:43.529ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
90812025-04-30T06:42:43.530ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90822025-04-30T06:42:43.533ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 28220 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = dfe977d8-2b8e-43a1-9741-5bb40bc63ad0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
90832025-04-30T06:42:43.535ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfe977d8-2b8e-43a1-9741-5bb40bc63ad0", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
90842025-04-30T06:42:43.545ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
90852025-04-30T06:42:43.545ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90862025-04-30T06:42:43.551ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90872025-04-30T06:42:43.551ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90882025-04-30T06:42:43.551ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90892025-04-30T06:42:43.552ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90902025-04-30T06:42:43.555ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1387293a-ea7e-42c7-a2ee-9eaf9e9990ab uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
90912025-04-30T06:42:43.555ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90922025-04-30T06:42:43.555ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1387293a-ea7e-42c7-a2ee-9eaf9e9990ab resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
90932025-04-30T06:42:43.555ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1387293a-ea7e-42c7-a2ee-9eaf9e9990ab roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
90942025-04-30T06:42:43.557ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1387293a-ea7e-42c7-a2ee-9eaf9e9990ab resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
90952025-04-30T06:42:43.557ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90962025-04-30T06:42:43.558ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90972025-04-30T06:42:43.558ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
90982025-04-30T06:42:43.560ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90992025-04-30T06:42:43.566ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91002025-04-30T06:42:43.568ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91012025-04-30T06:42:43.568ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91022025-04-30T06:42:43.568ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91032025-04-30T06:42:43.569ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 14582 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1387293a-ea7e-42c7-a2ee-9eaf9e9990ab response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
91042025-04-30T06:42:43.572ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "1387293a-ea7e-42c7-a2ee-9eaf9e9990ab", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91052025-04-30T06:42:43.572ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91062025-04-30T06:42:43.573ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
91072025-04-30T06:42:43.573ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
91082025-04-30T06:42:43.614ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
91092025-04-30T06:42:43.614ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
91102025-04-30T06:42:43.614ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 511.440873ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
91112025-04-30T06:42:43.615ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
91122025-04-30T06:42:43.615ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
91132025-04-30T06:42:43.615ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91142025-04-30T06:42:43.615ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91152025-04-30T06:42:43.615ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91162025-04-30T06:42:43.617ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91172025-04-30T06:42:43.619ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91182025-04-30T06:42:43.624ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
91192025-04-30T06:42:43.624ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91202025-04-30T06:42:43.624ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91212025-04-30T06:42:43.625ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91222025-04-30T06:42:43.630ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
91232025-04-30T06:42:43.632ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
91242025-04-30T06:42:43.632ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
91252025-04-30T06:42:43.633ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 93 f6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64671 req_id = bda20821-6ca2-48c8-9556-4f53577d165d
91262025-04-30T06:42:43.633ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37878,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64671 req_id = bda20821-6ca2-48c8-9556-4f53577d165d
91272025-04-30T06:42:43.634ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
91282025-04-30T06:42:43.634ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
91292025-04-30T06:42:43.634ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64671 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bda20821-6ca2-48c8-9556-4f53577d165d
91302025-04-30T06:42:43.634ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4228, tv_nsec: 433758317 } })
91312025-04-30T06:42:43.634ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
91322025-04-30T06:42:43.635ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 58 df 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e X...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63884 req_id = a645ebea-5328-4eaf-921b-0f801520a8bf
91332025-04-30T06:42:43.635ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22751,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63884 req_id = a645ebea-5328-4eaf-921b-0f801520a8bf
91342025-04-30T06:42:43.635ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
91352025-04-30T06:42:43.635ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63884 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a645ebea-5328-4eaf-921b-0f801520a8bf
91362025-04-30T06:42:43.723ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
91372025-04-30T06:42:43.724ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91382025-04-30T06:42:43.725ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91392025-04-30T06:42:43.725ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91402025-04-30T06:42:43.725ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
91412025-04-30T06:42:43.727ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91422025-04-30T06:42:43.728ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 03 8e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58995 req_id = eeb27b9c-ad9b-461c-be90-892592d027f1
91432025-04-30T06:42:43.728ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 910,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58995 req_id = eeb27b9c-ad9b-461c-be90-892592d027f1
91442025-04-30T06:42:43.728ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
91452025-04-30T06:42:43.728ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 910, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58995 req_id = eeb27b9c-ad9b-461c-be90-892592d027f1
91462025-04-30T06:42:43.735ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
91472025-04-30T06:42:43.735ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
91482025-04-30T06:42:43.735ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
91492025-04-30T06:42:43.736ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91502025-04-30T06:42:43.749ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91512025-04-30T06:42:43.749ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91522025-04-30T06:42:43.749ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
91532025-04-30T06:42:43.750ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91542025-04-30T06:42:43.753ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91552025-04-30T06:42:43.753ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
91562025-04-30T06:42:43.754ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91572025-04-30T06:42:43.754ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5df41508-f15f-44e6-9c0b-9ba3ca959ace", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91582025-04-30T06:42:43.754ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
91592025-04-30T06:42:43.756ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2f03ba3-8eff-4d7a-810c-3332e957eda7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91602025-04-30T06:42:43.757ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
91612025-04-30T06:42:43.763ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1ab7628-8a75-481f-b9ac-7ec80bb39727", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91622025-04-30T06:42:43.763ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
91632025-04-30T06:42:43.766ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1475109d-8256-4ee1-8341-d28e34baee81", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91642025-04-30T06:42:43.766ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
91652025-04-30T06:42:43.769ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62ebe458-4a84-4836-b38e-5fbb3a98785c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91662025-04-30T06:42:43.770ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
91672025-04-30T06:42:43.785ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa228318-865e-46f7-b2bd-ba59de8420b8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91682025-04-30T06:42:43.786ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
91692025-04-30T06:42:43.790ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "caa2bca9-ea98-47be-8433-2e549fd1f95f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91702025-04-30T06:42:43.790ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
91712025-04-30T06:42:43.795ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
91722025-04-30T06:42:43.796ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a8936ff-f4cc-4761-b209-0ceb3489329e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91732025-04-30T06:42:43.796ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
91742025-04-30T06:42:43.797ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2b3e79c-5675-4014-83fc-d203048b57a4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91752025-04-30T06:42:43.797ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
91762025-04-30T06:42:43.801ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8231302-af20-4e18-962f-07f3c1de6fd0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91772025-04-30T06:42:43.801ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
91782025-04-30T06:42:43.804ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a74a1a68-dee8-4e08-a741-a938030a1584", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91792025-04-30T06:42:43.804ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
91802025-04-30T06:42:43.806ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8421a19a-d264-47e2-aa73-b8b365557e57", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91812025-04-30T06:42:43.806ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
91822025-04-30T06:42:43.812ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a227752d-7100-4831-b87e-f92406164398", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91832025-04-30T06:42:43.812ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
91842025-04-30T06:42:43.817ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4b2c616-7feb-45a3-9736-b867da345061", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91852025-04-30T06:42:43.817ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
91862025-04-30T06:42:43.825ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0384870a-497f-4add-ab34-422d0a6ce5bb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91872025-04-30T06:42:43.825ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
91882025-04-30T06:42:43.835ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "247f1904-1a60-4d9c-9426-aa79a6174932", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91892025-04-30T06:42:43.835ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
91902025-04-30T06:42:43.842ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66f479e3-4d1f-4255-bb38-60b13ce1714b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91912025-04-30T06:42:43.842ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
91922025-04-30T06:42:43.845ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d7eb5d8-b722-4d67-a1db-00b4f95736ec", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
91932025-04-30T06:42:43.846ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
91942025-04-30T06:42:43.846ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91952025-04-30T06:42:43.846ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91962025-04-30T06:42:43.846ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91972025-04-30T06:42:43.846ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
91982025-04-30T06:42:43.847ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91992025-04-30T06:42:43.850ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92002025-04-30T06:42:43.859ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92012025-04-30T06:42:43.859ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92022025-04-30T06:42:43.859ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
92032025-04-30T06:42:43.860ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92042025-04-30T06:42:43.878ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "953e392d-eab6-4c4c-8058-2bbb5ba2790e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92052025-04-30T06:42:43.878ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
92062025-04-30T06:42:43.879ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92072025-04-30T06:42:43.879ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
92082025-04-30T06:42:43.880ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92092025-04-30T06:42:43.880ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18bf17fe-1e67-4ac7-97cc-035ae7d1e23c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92102025-04-30T06:42:43.881ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
92112025-04-30T06:42:43.882ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f271e44d-73c8-4ca1-a3a7-3740ded2069b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92122025-04-30T06:42:43.882ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
92132025-04-30T06:42:43.887ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "809f5580-6ef9-44b8-aa70-2374255e0b14", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92142025-04-30T06:42:43.887ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
92152025-04-30T06:42:43.888ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "389d223d-8266-49d0-b7b6-c71ed5a43dfc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92162025-04-30T06:42:43.889ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
92172025-04-30T06:42:43.890ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cef11fa-e0f9-4a87-b672-1631e8ed75f8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92182025-04-30T06:42:43.890ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
92192025-04-30T06:42:43.893ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceabb1e3-f76e-44d2-bd28-8fee66daaa69", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92202025-04-30T06:42:43.893ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
92212025-04-30T06:42:43.896ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "915d222b-a478-47f4-9342-187cc4013349", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92222025-04-30T06:42:43.896ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
92232025-04-30T06:42:43.898ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b1fbcc9-29c5-4934-ad31-2ac955056621", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92242025-04-30T06:42:43.898ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
92252025-04-30T06:42:43.901ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
92262025-04-30T06:42:43.901ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
92272025-04-30T06:42:43.902ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
92282025-04-30T06:42:43.905ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
92292025-04-30T06:42:43.912ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
92302025-04-30T06:42:43.913ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
92312025-04-30T06:42:43.918ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
92322025-04-30T06:42:43.918ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
92332025-04-30T06:42:43.919ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6d0932b-3b3d-4b4b-976e-9b4146cd7521", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92342025-04-30T06:42:43.919ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
92352025-04-30T06:42:43.921ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e11b0b67-0b8a-47ab-87c4-1cea31001ca4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92362025-04-30T06:42:43.921ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
92372025-04-30T06:42:43.926ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
92382025-04-30T06:42:43.929ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1719d009-ecc5-4933-9c65-fbbe0d751465", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92392025-04-30T06:42:43.929ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
92402025-04-30T06:42:43.930ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22d1e4a3-1a27-43af-ab47-8a9d053c8760", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92412025-04-30T06:42:43.930ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
92422025-04-30T06:42:43.931ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "919fa561-ad41-406a-80c7-bc71a744b951", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92432025-04-30T06:42:43.932ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
92442025-04-30T06:42:43.933ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a834888-a9e9-4e3d-8377-9d1f9a30b28a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92452025-04-30T06:42:43.933ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
92462025-04-30T06:42:43.934ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fcc4591-bdc6-430f-bc54-f1ea49d80b4e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92472025-04-30T06:42:43.934ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
92482025-04-30T06:42:43.936ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b350961-16b9-46ea-82a7-b51465ea9fba", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92492025-04-30T06:42:43.936ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
92502025-04-30T06:42:43.938ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1b662e1-bab7-4ff0-bff7-bfcda797af20", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92512025-04-30T06:42:43.938ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
92522025-04-30T06:42:43.942ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b758b5b-a869-450f-a0dc-a7f79def9a6f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92532025-04-30T06:42:43.942ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
92542025-04-30T06:42:43.947ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec4e51a6-ac1a-4134-b217-f20f9e1b7949", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92552025-04-30T06:42:43.947ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
92562025-04-30T06:42:43.949ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c8340b9-faac-4594-9807-9a145a8676c2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92572025-04-30T06:42:43.949ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
92582025-04-30T06:42:43.951ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7a9802e-4756-4e5a-9a5c-928b5c041b59", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92592025-04-30T06:42:43.951ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
92602025-04-30T06:42:43.954ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e58f7bb2-fde2-47a3-a3a0-271a50f2c0b2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92612025-04-30T06:42:43.954ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
92622025-04-30T06:42:43.955ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "337c4a5d-67ce-43f1-abd3-b8284b6b9845", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92632025-04-30T06:42:43.955ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
92642025-04-30T06:42:43.956ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "597dbd9e-69a8-4773-ba0f-6ecd3821f82c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92652025-04-30T06:42:43.956ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
92662025-04-30T06:42:43.957ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1703b6e-992c-4ec6-a638-3e707ca94ceb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92672025-04-30T06:42:43.957ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
92682025-04-30T06:42:43.958ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1af32f01-d9ea-4e78-9f79-641b96ef9bc4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:42 GMT"} })
92692025-04-30T06:42:43.959ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92702025-04-30T06:42:43.961ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92712025-04-30T06:42:43.961ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92722025-04-30T06:42:43.961ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92732025-04-30T06:42:43.962ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92742025-04-30T06:42:43.981ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92752025-04-30T06:42:43.981ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92762025-04-30T06:42:43.981ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92772025-04-30T06:42:43.981ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
92782025-04-30T06:42:43.985ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92792025-04-30T06:42:43.985ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92802025-04-30T06:42:43.991ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92812025-04-30T06:42:43.991ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92822025-04-30T06:42:43.992ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92832025-04-30T06:42:43.992ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92842025-04-30T06:42:43.992ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92852025-04-30T06:42:43.993ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92862025-04-30T06:42:43.997ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92872025-04-30T06:42:44.006ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92882025-04-30T06:42:44.006ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92892025-04-30T06:42:44.006ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
92902025-04-30T06:42:44.007ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92912025-04-30T06:42:44.007ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92922025-04-30T06:42:44.011ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92932025-04-30T06:42:44.011ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92942025-04-30T06:42:44.011ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92952025-04-30T06:42:44.011ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92962025-04-30T06:42:44.012ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
92972025-04-30T06:42:44.014ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92982025-04-30T06:42:44.014ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
92992025-04-30T06:42:44.014ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93002025-04-30T06:42:44.074ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
93012025-04-30T06:42:44.074ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
93022025-04-30T06:42:44.074ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:51469 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
93032025-04-30T06:42:44.076ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:51469 req_id = bf30ecda-1840-42b0-ad2b-8ae2dd661e9b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
93042025-04-30T06:42:44.076ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 557 local_addr = [::1]:62607 method = GET remote_addr = [::1]:51469 req_id = bf30ecda-1840-42b0-ad2b-8ae2dd661e9b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
93052025-04-30T06:42:44.077ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf30ecda-1840-42b0-ad2b-8ae2dd661e9b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93062025-04-30T06:42:44.080ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
93072025-04-30T06:42:44.080ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
93082025-04-30T06:42:44.080ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
93092025-04-30T06:42:44.082ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
93102025-04-30T06:42:44.082ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93112025-04-30T06:42:44.082ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93122025-04-30T06:42:44.082ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93132025-04-30T06:42:44.083ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93142025-04-30T06:42:44.087ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93152025-04-30T06:42:44.091ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93162025-04-30T06:42:44.091ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93172025-04-30T06:42:44.091ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93182025-04-30T06:42:44.092ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93192025-04-30T06:42:44.165ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
93202025-04-30T06:42:44.165ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
93212025-04-30T06:42:44.165ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 550.661026ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
93222025-04-30T06:42:44.166ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
93232025-04-30T06:42:44.166ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
93242025-04-30T06:42:44.166ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93252025-04-30T06:42:44.166ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93262025-04-30T06:42:44.166ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93272025-04-30T06:42:44.168ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93282025-04-30T06:42:44.168ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
93292025-04-30T06:42:44.171ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93302025-04-30T06:42:44.180ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93312025-04-30T06:42:44.180ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93322025-04-30T06:42:44.180ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93332025-04-30T06:42:44.181ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93342025-04-30T06:42:44.184ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
93352025-04-30T06:42:44.184ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
93362025-04-30T06:42:44.185ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a9 a1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64579 req_id = 25094628-b910-4cdf-9ec1-7fae7ef1fc58
93372025-04-30T06:42:44.185ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43425,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64579 req_id = 25094628-b910-4cdf-9ec1-7fae7ef1fc58
93382025-04-30T06:42:44.186ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
93392025-04-30T06:42:44.186ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93402025-04-30T06:42:44.186ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64579 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 25094628-b910-4cdf-9ec1-7fae7ef1fc58
93412025-04-30T06:42:44.187ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4228, tv_nsec: 986494498 } })
93422025-04-30T06:42:44.187ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
93432025-04-30T06:42:44.188ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c4 b9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63135 req_id = 08ae1a6f-3474-4bad-a442-9b5ac521a14d
93442025-04-30T06:42:44.188ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50361,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63135 req_id = 08ae1a6f-3474-4bad-a442-9b5ac521a14d
93452025-04-30T06:42:44.188ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93462025-04-30T06:42:44.188ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63135 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 08ae1a6f-3474-4bad-a442-9b5ac521a14d
93472025-04-30T06:42:44.252ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
93482025-04-30T06:42:44.260ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93492025-04-30T06:42:44.260ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93502025-04-30T06:42:44.260ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93512025-04-30T06:42:44.260ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
93522025-04-30T06:42:44.261ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93532025-04-30T06:42:44.262ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 85 08 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54146 req_id = 660f1a90-4732-493e-b8c4-6b572a67dcd5
93542025-04-30T06:42:44.262ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34056,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54146 req_id = 660f1a90-4732-493e-b8c4-6b572a67dcd5
93552025-04-30T06:42:44.262ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
93562025-04-30T06:42:44.262ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34056, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54146 req_id = 660f1a90-4732-493e-b8c4-6b572a67dcd5
93572025-04-30T06:42:44.265ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
93582025-04-30T06:42:44.265ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
93592025-04-30T06:42:44.265ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
93602025-04-30T06:42:44.265ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93612025-04-30T06:42:44.274ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93622025-04-30T06:42:44.274ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93632025-04-30T06:42:44.274ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
93642025-04-30T06:42:44.275ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93652025-04-30T06:42:44.280ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93662025-04-30T06:42:44.280ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
93672025-04-30T06:42:44.282ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93682025-04-30T06:42:44.290ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "88478948-5bdb-47f1-a67a-a014eed910b8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93692025-04-30T06:42:44.290ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
93702025-04-30T06:42:44.292ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4522310-9b7d-48f6-8dfb-dfed9cda7c2a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93712025-04-30T06:42:44.292ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
93722025-04-30T06:42:44.295ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa6932b2-2d67-4b1e-a615-1bea89781ac6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93732025-04-30T06:42:44.295ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
93742025-04-30T06:42:44.298ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a84601c7-7089-4751-a4c3-b807d9893931", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93752025-04-30T06:42:44.299ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
93762025-04-30T06:42:44.305ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aed765db-cec9-46ca-87f1-1d4235df8ee9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93772025-04-30T06:42:44.305ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
93782025-04-30T06:42:44.319ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40116e3e-470a-4f32-8842-751fa2ec99f7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93792025-04-30T06:42:44.319ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
93802025-04-30T06:42:44.321ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "391acd86-aaee-462f-9a38-baad0f10e268", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93812025-04-30T06:42:44.321ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
93822025-04-30T06:42:44.323ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "343d7c92-13ff-456f-82ed-62c172148ec0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93832025-04-30T06:42:44.324ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
93842025-04-30T06:42:44.324ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb6ed7ae-49f3-46ac-b777-97c2658eff96", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93852025-04-30T06:42:44.324ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
93862025-04-30T06:42:44.326ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17e22c23-9a7f-41c6-b235-7e34f3381be3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93872025-04-30T06:42:44.326ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
93882025-04-30T06:42:44.328ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfb29cec-26a0-4968-95fe-062d4fe90527", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93892025-04-30T06:42:44.330ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
93902025-04-30T06:42:44.330ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
93912025-04-30T06:42:44.332ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1670b83-ff3a-4773-88c6-2ccae67e32c8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93922025-04-30T06:42:44.333ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
93932025-04-30T06:42:44.335ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "682a0ed6-a486-4929-849f-81f5cb8c06d5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93942025-04-30T06:42:44.336ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
93952025-04-30T06:42:44.337ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91cfba8b-310f-45e7-975c-dffc9ec05f57", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93962025-04-30T06:42:44.338ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
93972025-04-30T06:42:44.340ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f946d424-e0a9-4626-a494-4636fe33a865", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
93982025-04-30T06:42:44.340ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
93992025-04-30T06:42:44.342ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d4b826a-cf36-4bed-80c2-dccb497534fe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94002025-04-30T06:42:44.342ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
94012025-04-30T06:42:44.345ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2256ffc-ce44-4f91-ac84-d273c4101a0b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94022025-04-30T06:42:44.345ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
94032025-04-30T06:42:44.347ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73856716-2462-4e64-b9f8-53e665931d24", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94042025-04-30T06:42:44.347ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
94052025-04-30T06:42:44.349ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5894f6e9-c578-4ac4-ac1d-b4fea72a9e30", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94062025-04-30T06:42:44.349ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
94072025-04-30T06:42:44.350ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f947071-6b5e-4407-b4c5-4daec49b2d9e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94082025-04-30T06:42:44.350ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
94092025-04-30T06:42:44.351ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d4506e6-ad9e-4d07-8255-33c24e88d47f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94102025-04-30T06:42:44.352ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
94112025-04-30T06:42:44.353ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0526e48-dac5-4b9a-887a-3b376cceef2a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94122025-04-30T06:42:44.353ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
94132025-04-30T06:42:44.354ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0598e77-228f-488b-8672-e984a5015cbb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94142025-04-30T06:42:44.354ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
94152025-04-30T06:42:44.356ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8928421-a6d1-422a-a440-940cc0b247e2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94162025-04-30T06:42:44.356ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
94172025-04-30T06:42:44.358ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e495cc77-4732-40d3-abc4-4f825b55b4db", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94182025-04-30T06:42:44.358ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
94192025-04-30T06:42:44.360ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe98573d-9711-4c7f-9109-39cb95b30ad9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94202025-04-30T06:42:44.360ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
94212025-04-30T06:42:44.361ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04c92b64-5934-432e-9bb0-7b99d21358d9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94222025-04-30T06:42:44.361ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
94232025-04-30T06:42:44.364ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd5104a-72ff-4792-9a31-a69f88a1e65b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94242025-04-30T06:42:44.364ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
94252025-04-30T06:42:44.366ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "858641f7-7967-47e0-bcc5-439c32083548", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94262025-04-30T06:42:44.366ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
94272025-04-30T06:42:44.368ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc9e2b36-ed43-4d35-8b3b-b3158a1bc03c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94282025-04-30T06:42:44.370ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
94292025-04-30T06:42:44.372ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e57d817-b049-46a6-813d-07257e105f5d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94302025-04-30T06:42:44.372ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
94312025-04-30T06:42:44.373ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9603c34a-615c-4cb3-97a7-6ef0688e5750", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94322025-04-30T06:42:44.373ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
94332025-04-30T06:42:44.374ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44e1d92f-f9b9-4f2b-8f84-f4eee874359a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94342025-04-30T06:42:44.374ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
94352025-04-30T06:42:44.375ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72703a68-dde4-4094-a2c3-67923dd00044", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94362025-04-30T06:42:44.375ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
94372025-04-30T06:42:44.377ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1b60ab1-ac33-41b8-8669-7821d459ad35", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94382025-04-30T06:42:44.377ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
94392025-04-30T06:42:44.379ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb5d8949-fc13-43a7-8193-47111920f204", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94402025-04-30T06:42:44.379ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
94412025-04-30T06:42:44.381ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94422025-04-30T06:42:44.381ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94432025-04-30T06:42:44.381ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94442025-04-30T06:42:44.381ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
94452025-04-30T06:42:44.382ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94462025-04-30T06:42:44.383ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89480842-78b5-4a63-8d47-7aaf530c0c49", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94472025-04-30T06:42:44.383ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
94482025-04-30T06:42:44.385ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94492025-04-30T06:42:44.390ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94502025-04-30T06:42:44.390ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94512025-04-30T06:42:44.390ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
94522025-04-30T06:42:44.390ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94532025-04-30T06:42:44.393ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66d53448-4632-4969-b757-a3e666911238", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94542025-04-30T06:42:44.393ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
94552025-04-30T06:42:44.394ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94562025-04-30T06:42:44.394ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
94572025-04-30T06:42:44.395ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94582025-04-30T06:42:44.400ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "def27a14-1f1a-4ac5-8575-d72c002ae20a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94592025-04-30T06:42:44.400ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
94602025-04-30T06:42:44.403ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ed7db8-4f92-4289-9766-d5278f16edf4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94612025-04-30T06:42:44.403ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
94622025-04-30T06:42:44.408ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d2091c1-50c8-41c8-a8dd-bf6fd57a2298", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94632025-04-30T06:42:44.408ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
94642025-04-30T06:42:44.411ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7694f0a1-6d68-4e39-8a2e-615feb8932d9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94652025-04-30T06:42:44.411ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
94662025-04-30T06:42:44.416ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10ea70aa-ab84-4720-979e-718feece5261", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94672025-04-30T06:42:44.416ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
94682025-04-30T06:42:44.417ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26dabe2f-5ec2-47fd-b839-1e54909117af", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94692025-04-30T06:42:44.417ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
94702025-04-30T06:42:44.420ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9f570af-c242-42b7-8989-04d055ac71e7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
94712025-04-30T06:42:44.420ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94722025-04-30T06:42:44.423ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94732025-04-30T06:42:44.423ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94742025-04-30T06:42:44.423ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94752025-04-30T06:42:44.423ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94762025-04-30T06:42:44.425ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94772025-04-30T06:42:44.429ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94782025-04-30T06:42:44.429ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94792025-04-30T06:42:44.429ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94802025-04-30T06:42:44.429ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94812025-04-30T06:42:44.429ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94822025-04-30T06:42:44.429ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94832025-04-30T06:42:44.449ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
94842025-04-30T06:42:44.449ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94852025-04-30T06:42:44.457ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94862025-04-30T06:42:44.458ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94872025-04-30T06:42:44.458ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94882025-04-30T06:42:44.458ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94892025-04-30T06:42:44.558ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
94902025-04-30T06:42:44.558ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
94912025-04-30T06:42:44.559ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94922025-04-30T06:42:44.559ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94932025-04-30T06:42:44.559ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94942025-04-30T06:42:44.559ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
94952025-04-30T06:42:44.561ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94962025-04-30T06:42:44.561ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:51214 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
94972025-04-30T06:42:44.561ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
94982025-04-30T06:42:44.561ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
94992025-04-30T06:42:44.562ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:51214 req_id = 5d55b0dc-22a6-40a4-8214-367af818dfbc sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
95002025-04-30T06:42:44.565ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 2833 local_addr = [::1]:62607 method = GET remote_addr = [::1]:51214 req_id = 5d55b0dc-22a6-40a4-8214-367af818dfbc response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
95012025-04-30T06:42:44.565ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95022025-04-30T06:42:44.566ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95032025-04-30T06:42:44.566ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95042025-04-30T06:42:44.566ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95052025-04-30T06:42:44.566ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d55b0dc-22a6-40a4-8214-367af818dfbc", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95062025-04-30T06:42:44.572ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95072025-04-30T06:42:44.572ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95082025-04-30T06:42:44.572ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
95092025-04-30T06:42:44.574ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95102025-04-30T06:42:44.574ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95112025-04-30T06:42:44.574ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95122025-04-30T06:42:44.574ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95132025-04-30T06:42:44.575ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95142025-04-30T06:42:44.575ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95152025-04-30T06:42:44.580ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95162025-04-30T06:42:44.580ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95172025-04-30T06:42:44.580ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
95182025-04-30T06:42:44.581ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95192025-04-30T06:42:44.585ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95202025-04-30T06:42:44.590ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95212025-04-30T06:42:44.591ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95222025-04-30T06:42:44.591ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95232025-04-30T06:42:44.592ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95242025-04-30T06:42:44.592ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95252025-04-30T06:42:44.594ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
95262025-04-30T06:42:44.594ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95272025-04-30T06:42:44.672ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
95282025-04-30T06:42:44.672ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
95292025-04-30T06:42:44.673ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 506.979374ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
95302025-04-30T06:42:44.673ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
95312025-04-30T06:42:44.673ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95322025-04-30T06:42:44.673ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95332025-04-30T06:42:44.673ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95342025-04-30T06:42:44.673ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95352025-04-30T06:42:44.674ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95362025-04-30T06:42:44.675ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
95372025-04-30T06:42:44.676ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95382025-04-30T06:42:44.682ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95392025-04-30T06:42:44.682ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95402025-04-30T06:42:44.682ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95412025-04-30T06:42:44.683ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95422025-04-30T06:42:44.688ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
95432025-04-30T06:42:44.688ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
95442025-04-30T06:42:44.689ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fb 43 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .C..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62418 req_id = 0bf8b6b9-73a6-4740-9a4b-8594c3adcd93
95452025-04-30T06:42:44.689ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64323,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62418 req_id = 0bf8b6b9-73a6-4740-9a4b-8594c3adcd93
95462025-04-30T06:42:44.690ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
95472025-04-30T06:42:44.690ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95482025-04-30T06:42:44.690ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62418 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0bf8b6b9-73a6-4740-9a4b-8594c3adcd93
95492025-04-30T06:42:44.690ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4229, tv_nsec: 489782667 } })
95502025-04-30T06:42:44.691ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
95512025-04-30T06:42:44.691ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 26 f5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e &...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58368 req_id = cde315f8-d143-448f-bcab-7f300c505512
95522025-04-30T06:42:44.691ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9973,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58368 req_id = cde315f8-d143-448f-bcab-7f300c505512
95532025-04-30T06:42:44.691ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95542025-04-30T06:42:44.691ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58368 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cde315f8-d143-448f-bcab-7f300c505512
95552025-04-30T06:42:44.767ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
95562025-04-30T06:42:44.768ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95572025-04-30T06:42:44.768ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95582025-04-30T06:42:44.768ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95592025-04-30T06:42:44.769ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
95602025-04-30T06:42:44.773ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95612025-04-30T06:42:44.773ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 59 b3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Y............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54955 req_id = 879af9ff-3bac-4080-9293-55b945153f7e
95622025-04-30T06:42:44.774ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22963,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54955 req_id = 879af9ff-3bac-4080-9293-55b945153f7e
95632025-04-30T06:42:44.774ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
95642025-04-30T06:42:44.774ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22963, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54955 req_id = 879af9ff-3bac-4080-9293-55b945153f7e
95652025-04-30T06:42:44.775ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
95662025-04-30T06:42:44.775ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
95672025-04-30T06:42:44.775ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
95682025-04-30T06:42:44.776ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95692025-04-30T06:42:44.788ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95702025-04-30T06:42:44.788ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95712025-04-30T06:42:44.788ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
95722025-04-30T06:42:44.789ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95732025-04-30T06:42:44.794ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ba8cf18-b487-4a7d-bc87-569dd6b4fc53", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95742025-04-30T06:42:44.794ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
95752025-04-30T06:42:44.797ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95762025-04-30T06:42:44.797ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
95772025-04-30T06:42:44.797ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95782025-04-30T06:42:44.802ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fd5e108-89bc-4f55-ae81-569b195fd0f6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95792025-04-30T06:42:44.802ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
95802025-04-30T06:42:44.808ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29706c7f-d852-4770-969e-99f22ed723f3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95812025-04-30T06:42:44.808ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
95822025-04-30T06:42:44.815ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ee2f09b-a65f-4852-b49d-31760c5230ff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95832025-04-30T06:42:44.815ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
95842025-04-30T06:42:44.823ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e16a8352-9a29-4467-afd5-e9fd25588e0d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95852025-04-30T06:42:44.824ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
95862025-04-30T06:42:44.826ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b895ca7-5d76-4774-9348-fb4926c07c87", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95872025-04-30T06:42:44.827ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
95882025-04-30T06:42:44.836ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3f6e9fc-bd3c-41ec-a68a-c8069ef508f2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95892025-04-30T06:42:44.836ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
95902025-04-30T06:42:44.844ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a8aded-068c-4965-b2df-86bf480f340b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95912025-04-30T06:42:44.844ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
95922025-04-30T06:42:44.845ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
95932025-04-30T06:42:44.845ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58749603-d92a-4ddf-bb46-9b766a8f3f1e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95942025-04-30T06:42:44.846ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
95952025-04-30T06:42:44.848ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75d14057-ec6b-41a9-8bee-e6a376a6b586", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95962025-04-30T06:42:44.849ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
95972025-04-30T06:42:44.850ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32cabb52-f9e2-493f-ab97-adf8e0c34135", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
95982025-04-30T06:42:44.850ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
95992025-04-30T06:42:44.851ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c550e338-9764-486d-ba8f-bf50ebfa51f7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96002025-04-30T06:42:44.851ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
96012025-04-30T06:42:44.853ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00394a2d-d9ae-4a84-88bf-f800e16b18c2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96022025-04-30T06:42:44.854ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
96032025-04-30T06:42:44.858ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06bbeb52-7e33-4b59-8598-0fe0eb1b3b3e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96042025-04-30T06:42:44.858ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
96052025-04-30T06:42:44.862ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "945d4f7a-fafd-4420-ad62-8a5b3a41ad9c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96062025-04-30T06:42:44.862ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
96072025-04-30T06:42:44.865ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9da2bdd2-ea41-4a7f-8955-5b153c25ca9f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96082025-04-30T06:42:44.865ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
96092025-04-30T06:42:44.867ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c4dd8d3-bfd4-4d72-b147-abba4fb7d7eb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96102025-04-30T06:42:44.867ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
96112025-04-30T06:42:44.869ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47e97491-69bf-4e21-9090-2296d0fda296", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96122025-04-30T06:42:44.869ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
96132025-04-30T06:42:44.874ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "057b10ef-d348-476d-9c52-c3b68ed4be85", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96142025-04-30T06:42:44.874ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
96152025-04-30T06:42:44.874ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebc510e3-93ca-46b4-94e9-84e944582c7a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96162025-04-30T06:42:44.874ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
96172025-04-30T06:42:44.876ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b053013-9b80-420b-8887-de6c8c590c95", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96182025-04-30T06:42:44.877ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
96192025-04-30T06:42:44.878ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd1c2cab-6d7e-42a8-ab17-a4e20f5da2fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96202025-04-30T06:42:44.878ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
96212025-04-30T06:42:44.879ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dec5c46-5e05-4e39-b603-f16d819eec26", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96222025-04-30T06:42:44.879ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
96232025-04-30T06:42:44.880ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6197e29-667c-4d52-982b-1a98571d4753", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96242025-04-30T06:42:44.881ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
96252025-04-30T06:42:44.883ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d55b20b-060d-447a-a068-1431b2a4440f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96262025-04-30T06:42:44.883ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
96272025-04-30T06:42:44.885ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e1bbf13-d2e5-4f0d-b317-ccad19ab1d86", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96282025-04-30T06:42:44.885ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
96292025-04-30T06:42:44.888ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab88e02f-41ec-4615-b426-49d9d44e6537", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96302025-04-30T06:42:44.888ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
96312025-04-30T06:42:44.893ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f5990ca-2552-4a6b-bf2f-6d190997270c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96322025-04-30T06:42:44.893ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
96332025-04-30T06:42:44.896ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96342025-04-30T06:42:44.896ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96352025-04-30T06:42:44.896ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96362025-04-30T06:42:44.896ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
96372025-04-30T06:42:44.896ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96382025-04-30T06:42:44.897ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fe52e27-71cf-4574-83a8-9728b4cfab41", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96392025-04-30T06:42:44.897ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
96402025-04-30T06:42:44.899ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96412025-04-30T06:42:44.909ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96422025-04-30T06:42:44.909ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96432025-04-30T06:42:44.909ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
96442025-04-30T06:42:44.910ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96452025-04-30T06:42:44.914ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
96462025-04-30T06:42:44.914ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
96472025-04-30T06:42:44.915ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
96482025-04-30T06:42:44.916ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
96492025-04-30T06:42:44.916ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96502025-04-30T06:42:44.916ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
96512025-04-30T06:42:44.917ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96522025-04-30T06:42:44.920ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad74a4a-1cd2-487c-941d-990f60b940d4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96532025-04-30T06:42:44.920ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
96542025-04-30T06:42:44.922ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
96552025-04-30T06:42:44.922ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
96562025-04-30T06:42:44.923ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97c3e49f-d35e-4734-bf73-ad4048b938ea", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96572025-04-30T06:42:44.923ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
96582025-04-30T06:42:44.925ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8654886-1c5a-434f-95e8-4ef959a35e32", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96592025-04-30T06:42:44.925ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
96602025-04-30T06:42:44.927ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
96612025-04-30T06:42:44.927ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
96622025-04-30T06:42:44.929ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb187872-bd52-456e-a737-ae4048a261be", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96632025-04-30T06:42:44.929ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
96642025-04-30T06:42:44.931ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe5e0f9a-4407-472f-8c7a-18c1402ee805", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96652025-04-30T06:42:44.931ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
96662025-04-30T06:42:44.933ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c35a9e41-5503-40aa-ba9b-bd64ef528976", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96672025-04-30T06:42:44.933ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
96682025-04-30T06:42:44.936ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10019a5e-ad95-463f-bd44-b4c2c669081f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96692025-04-30T06:42:44.936ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
96702025-04-30T06:42:44.939ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8952740-5cea-4234-9a06-49834626346c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96712025-04-30T06:42:44.939ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
96722025-04-30T06:42:44.942ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da235b5e-9cb7-485a-be35-1a4ede69016d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96732025-04-30T06:42:44.942ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
96742025-04-30T06:42:44.945ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fcaa5e0-52df-4132-8ade-fc5ec8395122", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96752025-04-30T06:42:44.946ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
96762025-04-30T06:42:44.952ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
96772025-04-30T06:42:44.963ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f657552-22a1-4a9c-b0a8-863ed9fe100e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96782025-04-30T06:42:44.964ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
96792025-04-30T06:42:44.968ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10c5c6e6-4e2b-48d5-a832-1ff35a0b8b99", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96802025-04-30T06:42:44.975ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
96812025-04-30T06:42:44.980ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad278edc-4529-427d-83d6-f5868d7688ab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96822025-04-30T06:42:44.980ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
96832025-04-30T06:42:44.991ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7c6b396-6de4-4b2b-a71b-47acbaf181b6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96842025-04-30T06:42:44.991ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
96852025-04-30T06:42:44.993ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37ec5559-1855-4999-b5aa-1fe82c8ea95e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96862025-04-30T06:42:44.994ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
96872025-04-30T06:42:44.996ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13ef4f8b-ff7a-4bb9-8266-e1feee87e3a2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:44 GMT"} })
96882025-04-30T06:42:44.996ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96892025-04-30T06:42:44.996ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96902025-04-30T06:42:44.996ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96912025-04-30T06:42:44.996ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96922025-04-30T06:42:44.998ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96932025-04-30T06:42:45.005ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96942025-04-30T06:42:45.009ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96952025-04-30T06:42:45.009ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96962025-04-30T06:42:45.009ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96972025-04-30T06:42:45.009ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96982025-04-30T06:42:45.009ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96992025-04-30T06:42:45.009ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97002025-04-30T06:42:45.010ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97012025-04-30T06:42:45.010ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97022025-04-30T06:42:45.010ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97032025-04-30T06:42:45.010ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
97042025-04-30T06:42:45.011ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97052025-04-30T06:42:45.014ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97062025-04-30T06:42:45.020ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97072025-04-30T06:42:45.020ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97082025-04-30T06:42:45.020ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
97092025-04-30T06:42:45.021ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97102025-04-30T06:42:45.021ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97112025-04-30T06:42:45.025ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97122025-04-30T06:42:45.025ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97132025-04-30T06:42:45.025ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97142025-04-30T06:42:45.026ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97152025-04-30T06:42:45.028ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97162025-04-30T06:42:45.028ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
97172025-04-30T06:42:45.028ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97182025-04-30T06:42:45.029ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
97192025-04-30T06:42:45.098ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
97202025-04-30T06:42:45.098ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
97212025-04-30T06:42:45.098ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:53632 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
97222025-04-30T06:42:45.101ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:53632 req_id = bf149b5c-7320-4916-871b-f041b7796a12 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97232025-04-30T06:42:45.102ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 773 local_addr = [::1]:62607 method = GET remote_addr = [::1]:53632 req_id = bf149b5c-7320-4916-871b-f041b7796a12 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97242025-04-30T06:42:45.104ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf149b5c-7320-4916-871b-f041b7796a12", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
97252025-04-30T06:42:45.117ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97262025-04-30T06:42:45.117ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97272025-04-30T06:42:45.117ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
97282025-04-30T06:42:45.120ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97292025-04-30T06:42:45.120ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97302025-04-30T06:42:45.120ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97312025-04-30T06:42:45.121ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97322025-04-30T06:42:45.121ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97332025-04-30T06:42:45.134ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97342025-04-30T06:42:45.138ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97352025-04-30T06:42:45.138ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97362025-04-30T06:42:45.138ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97372025-04-30T06:42:45.139ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97382025-04-30T06:42:45.185ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
97392025-04-30T06:42:45.194ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
97402025-04-30T06:42:45.196ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
97412025-04-30T06:42:45.196ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 523.395621ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
97422025-04-30T06:42:45.196ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
97432025-04-30T06:42:45.196ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97442025-04-30T06:42:45.196ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97452025-04-30T06:42:45.196ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97462025-04-30T06:42:45.196ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97472025-04-30T06:42:45.197ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97482025-04-30T06:42:45.199ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97492025-04-30T06:42:45.203ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97502025-04-30T06:42:45.203ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97512025-04-30T06:42:45.203ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97522025-04-30T06:42:45.203ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97532025-04-30T06:42:45.207ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
97542025-04-30T06:42:45.207ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
97552025-04-30T06:42:45.208ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 77 0b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 w............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56108 req_id = ff4b886d-1e45-4996-9423-3fc14a0e89ab
97562025-04-30T06:42:45.208ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30475,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56108 req_id = ff4b886d-1e45-4996-9423-3fc14a0e89ab
97572025-04-30T06:42:45.208ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
97582025-04-30T06:42:45.208ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97592025-04-30T06:42:45.208ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56108 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ff4b886d-1e45-4996-9423-3fc14a0e89ab
97602025-04-30T06:42:45.209ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4230, tv_nsec: 8147655 } })
97612025-04-30T06:42:45.209ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
97622025-04-30T06:42:45.209ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 73 50 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e sP..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51965 req_id = 0f6ee688-6984-40e2-abc8-ada0e684d2e9
97632025-04-30T06:42:45.209ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29520,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51965 req_id = 0f6ee688-6984-40e2-abc8-ada0e684d2e9
97642025-04-30T06:42:45.209ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97652025-04-30T06:42:45.209ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51965 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0f6ee688-6984-40e2-abc8-ada0e684d2e9
97662025-04-30T06:42:45.266ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
97672025-04-30T06:42:45.266ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97682025-04-30T06:42:45.266ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97692025-04-30T06:42:45.266ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97702025-04-30T06:42:45.266ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
97712025-04-30T06:42:45.267ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97722025-04-30T06:42:45.268ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6f a3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c o............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64012 req_id = f96ceb71-9b4e-4f3e-874a-3ef6a9614e68
97732025-04-30T06:42:45.268ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28579,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64012 req_id = f96ceb71-9b4e-4f3e-874a-3ef6a9614e68
97742025-04-30T06:42:45.268ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
97752025-04-30T06:42:45.268ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28579, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64012 req_id = f96ceb71-9b4e-4f3e-874a-3ef6a9614e68
97762025-04-30T06:42:45.268ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
97772025-04-30T06:42:45.268ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
97782025-04-30T06:42:45.268ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
97792025-04-30T06:42:45.271ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97802025-04-30T06:42:45.278ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97812025-04-30T06:42:45.278ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97822025-04-30T06:42:45.278ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
97832025-04-30T06:42:45.279ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97842025-04-30T06:42:45.279ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e0815b1-eeda-414a-bdd1-819d6328dfb9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
97852025-04-30T06:42:45.280ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
97862025-04-30T06:42:45.283ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97872025-04-30T06:42:45.283ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
97882025-04-30T06:42:45.284ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97892025-04-30T06:42:45.284ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b3c596d-e7dc-48d3-af48-bdded4f69e56", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
97902025-04-30T06:42:45.284ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
97912025-04-30T06:42:45.286ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67d8e515-2412-4cdb-82c4-8b34c6d3a28e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
97922025-04-30T06:42:45.286ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
97932025-04-30T06:42:45.288ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfe4ac59-4e62-4e4c-a17a-5f1c73e7d094", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
97942025-04-30T06:42:45.288ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
97952025-04-30T06:42:45.292ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2f7271d-b2ce-4315-8a15-e2136ea10bd3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
97962025-04-30T06:42:45.292ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
97972025-04-30T06:42:45.295ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f50187d2-7438-41aa-b4b2-2da88c3a7366", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
97982025-04-30T06:42:45.295ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
97992025-04-30T06:42:45.298ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae6a4ca-3da8-46c6-9b79-cf030258bb76", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98002025-04-30T06:42:45.298ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
98012025-04-30T06:42:45.301ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48dc0957-20ed-4218-85e1-0a805a2294c5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98022025-04-30T06:42:45.301ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
98032025-04-30T06:42:45.302ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f96a136d-2940-44c7-9d42-6885e6b52cd3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98042025-04-30T06:42:45.302ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
98052025-04-30T06:42:45.304ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bbd7f82-ab0d-4eff-be7f-0fb05e3d61cc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98062025-04-30T06:42:45.305ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
98072025-04-30T06:42:45.306ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94c9b442-4fcc-49ce-a46d-c60be7c31657", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98082025-04-30T06:42:45.307ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
98092025-04-30T06:42:45.308ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13252b7c-315a-4599-8699-fba2eb86da38", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98102025-04-30T06:42:45.308ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
98112025-04-30T06:42:45.310ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f1df2cf-8b74-4d46-b386-0eb62a2cb99d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98122025-04-30T06:42:45.310ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
98132025-04-30T06:42:45.312ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ecf3ff9-eb68-4ef7-a11a-9cc400b9e542", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98142025-04-30T06:42:45.312ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
98152025-04-30T06:42:45.314ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13ad9ab7-927d-4dcd-94ce-bd9c3de67310", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98162025-04-30T06:42:45.315ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
98172025-04-30T06:42:45.319ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10204e25-cc69-47ed-8b44-9207257c968d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98182025-04-30T06:42:45.319ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
98192025-04-30T06:42:45.336ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
98202025-04-30T06:42:45.339ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "972d8f4f-cc47-4ff9-9e65-1ee254114efe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98212025-04-30T06:42:45.339ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
98222025-04-30T06:42:45.344ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "384b97ac-b2d9-4c06-b26c-37e530116c5e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98232025-04-30T06:42:45.345ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
98242025-04-30T06:42:45.347ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1964c905-9cdb-4e19-b659-5c4da4882d72", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98252025-04-30T06:42:45.347ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
98262025-04-30T06:42:45.348ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77c6ac1d-ca47-4b7e-99d6-618fe91c9010", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98272025-04-30T06:42:45.348ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
98282025-04-30T06:42:45.350ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c1ebac3-090e-481a-a855-5203b89d87e5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98292025-04-30T06:42:45.350ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
98302025-04-30T06:42:45.351ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "daeef27b-ebd9-496e-b88e-8a4a13eb7ebe", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98312025-04-30T06:42:45.351ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
98322025-04-30T06:42:45.352ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7575e569-086c-40a5-b932-791f5ae0164a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98332025-04-30T06:42:45.352ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
98342025-04-30T06:42:45.354ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e16662e-6fe0-4997-85b5-93ce0bb5ecc3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98352025-04-30T06:42:45.354ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
98362025-04-30T06:42:45.356ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d4b08ff-5dc8-49f0-8bb9-6e42cbb683ca", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98372025-04-30T06:42:45.356ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
98382025-04-30T06:42:45.358ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f0b0258-5c35-4e67-96a4-aa04ff3f1037", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98392025-04-30T06:42:45.358ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
98402025-04-30T06:42:45.360ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a966337-6fb5-4a90-a63e-367d9e2a8df8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98412025-04-30T06:42:45.360ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
98422025-04-30T06:42:45.362ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "508ad363-aac2-4259-ae87-016d531aca63", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98432025-04-30T06:42:45.363ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
98442025-04-30T06:42:45.364ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4e67816-40fa-4d51-9b70-7cd36c580de0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98452025-04-30T06:42:45.364ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
98462025-04-30T06:42:45.366ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "612e9258-0e4c-49f8-9eb8-d62145cc4428", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98472025-04-30T06:42:45.366ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
98482025-04-30T06:42:45.367ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22717407-c1f7-4b54-bde7-7767f029757e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98492025-04-30T06:42:45.367ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
98502025-04-30T06:42:45.368ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06edc7b8-bed9-4081-8de6-e50fff650de5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98512025-04-30T06:42:45.368ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
98522025-04-30T06:42:45.369ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19fa7e32-6447-4da9-9315-ee1afb6e920a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98532025-04-30T06:42:45.370ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
98542025-04-30T06:42:45.371ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6281a64f-0155-4a1f-afbe-e1209148c761", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98552025-04-30T06:42:45.371ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
98562025-04-30T06:42:45.373ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56bff44e-4ff4-46d1-9c02-7538216457e2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98572025-04-30T06:42:45.373ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
98582025-04-30T06:42:45.374ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b1bb8a1-2b35-4b51-929e-734ba008eb11", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98592025-04-30T06:42:45.375ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
98602025-04-30T06:42:45.376ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be4fd7c8-aeff-4af4-8311-5c1a2ef8bb55", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98612025-04-30T06:42:45.376ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
98622025-04-30T06:42:45.383ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f458a8ef-272f-4432-9a20-9df9d9b25f6b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98632025-04-30T06:42:45.383ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
98642025-04-30T06:42:45.385ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de28341b-a8fd-46c0-96c8-d21be92ebf62", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98652025-04-30T06:42:45.386ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
98662025-04-30T06:42:45.390ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98672025-04-30T06:42:45.390ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98682025-04-30T06:42:45.390ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98692025-04-30T06:42:45.390ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
98702025-04-30T06:42:45.390ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98712025-04-30T06:42:45.393ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98722025-04-30T06:42:45.403ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98732025-04-30T06:42:45.403ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98742025-04-30T06:42:45.404ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
98752025-04-30T06:42:45.404ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98762025-04-30T06:42:45.405ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bf59bda-59b0-4ca4-a43d-2f98f02204be", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98772025-04-30T06:42:45.405ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
98782025-04-30T06:42:45.408ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98792025-04-30T06:42:45.409ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
98802025-04-30T06:42:45.409ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98812025-04-30T06:42:45.412ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "337c093c-d446-45dd-96d6-479bca8061a5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98822025-04-30T06:42:45.413ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
98832025-04-30T06:42:45.415ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09cf7c6f-69c5-48bc-ae4c-fef562066991", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98842025-04-30T06:42:45.416ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
98852025-04-30T06:42:45.418ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59de16e1-dd85-46e6-889f-75a596bcf620", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98862025-04-30T06:42:45.418ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
98872025-04-30T06:42:45.421ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ade4c430-0b45-4977-8d58-97244177f012", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98882025-04-30T06:42:45.421ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
98892025-04-30T06:42:45.422ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a774ffeb-692e-4818-b250-bf3f809bf953", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
98902025-04-30T06:42:45.422ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98912025-04-30T06:42:45.423ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98922025-04-30T06:42:45.423ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98932025-04-30T06:42:45.423ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98942025-04-30T06:42:45.423ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98952025-04-30T06:42:45.426ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98962025-04-30T06:42:45.429ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98972025-04-30T06:42:45.429ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98982025-04-30T06:42:45.430ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98992025-04-30T06:42:45.430ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99002025-04-30T06:42:45.430ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99012025-04-30T06:42:45.431ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99022025-04-30T06:42:45.432ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99032025-04-30T06:42:45.435ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99042025-04-30T06:42:45.436ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99052025-04-30T06:42:45.437ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99062025-04-30T06:42:45.437ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99072025-04-30T06:42:45.531ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
99082025-04-30T06:42:45.531ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
99092025-04-30T06:42:45.532ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:43366 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
99102025-04-30T06:42:45.533ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
99112025-04-30T06:42:45.533ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99122025-04-30T06:42:45.533ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9d25dbc6-aaa9-4435-8fcf-edfd578f99fa uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99132025-04-30T06:42:45.534ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:43366 req_id = 507ae0f8-5d15-4faa-befd-d43d042bc503 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99142025-04-30T06:42:45.534ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9d25dbc6-aaa9-4435-8fcf-edfd578f99fa resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99152025-04-30T06:42:45.534ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9d25dbc6-aaa9-4435-8fcf-edfd578f99fa roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99162025-04-30T06:42:45.535ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9d25dbc6-aaa9-4435-8fcf-edfd578f99fa resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99172025-04-30T06:42:45.535ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1874 local_addr = [::1]:62607 method = GET remote_addr = [::1]:43366 req_id = 507ae0f8-5d15-4faa-befd-d43d042bc503 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99182025-04-30T06:42:45.537ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "507ae0f8-5d15-4faa-befd-d43d042bc503", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
99192025-04-30T06:42:45.539ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99202025-04-30T06:42:45.539ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99212025-04-30T06:42:45.539ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
99222025-04-30T06:42:45.541ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99232025-04-30T06:42:45.541ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99242025-04-30T06:42:45.541ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99252025-04-30T06:42:45.541ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99262025-04-30T06:42:45.541ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99272025-04-30T06:42:45.542ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
99282025-04-30T06:42:45.544ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 10995 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 9d25dbc6-aaa9-4435-8fcf-edfd578f99fa response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99292025-04-30T06:42:45.545ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d25dbc6-aaa9-4435-8fcf-edfd578f99fa", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
99302025-04-30T06:42:45.545ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
99312025-04-30T06:42:45.546ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99322025-04-30T06:42:45.550ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99332025-04-30T06:42:45.550ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99342025-04-30T06:42:45.550ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99352025-04-30T06:42:45.551ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99362025-04-30T06:42:45.552ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8d93eb7e-75a8-4662-93b4-bb551686aeab uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
99372025-04-30T06:42:45.552ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99382025-04-30T06:42:45.553ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8d93eb7e-75a8-4662-93b4-bb551686aeab resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
99392025-04-30T06:42:45.553ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8d93eb7e-75a8-4662-93b4-bb551686aeab roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
99402025-04-30T06:42:45.553ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8d93eb7e-75a8-4662-93b4-bb551686aeab resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
99412025-04-30T06:42:45.553ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99422025-04-30T06:42:45.556ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99432025-04-30T06:42:45.556ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99442025-04-30T06:42:45.557ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99452025-04-30T06:42:45.557ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99462025-04-30T06:42:45.558ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 5359 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 8d93eb7e-75a8-4662-93b4-bb551686aeab response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
99472025-04-30T06:42:45.558ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d93eb7e-75a8-4662-93b4-bb551686aeab", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
99482025-04-30T06:42:45.559ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99492025-04-30T06:42:45.559ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
99502025-04-30T06:42:45.559ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
99512025-04-30T06:42:45.583ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
99522025-04-30T06:42:45.583ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
99532025-04-30T06:42:45.583ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 387.182727ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
99542025-04-30T06:42:45.583ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
99552025-04-30T06:42:45.583ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99562025-04-30T06:42:45.583ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99572025-04-30T06:42:45.583ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99582025-04-30T06:42:45.584ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99592025-04-30T06:42:45.584ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99602025-04-30T06:42:45.590ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99612025-04-30T06:42:45.594ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99622025-04-30T06:42:45.594ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99632025-04-30T06:42:45.594ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99642025-04-30T06:42:45.595ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99652025-04-30T06:42:45.596ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99662025-04-30T06:42:45.596ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99672025-04-30T06:42:45.596ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99682025-04-30T06:42:45.596ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
99692025-04-30T06:42:45.598ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99702025-04-30T06:42:45.601ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99712025-04-30T06:42:45.613ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99722025-04-30T06:42:45.614ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99732025-04-30T06:42:45.614ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
99742025-04-30T06:42:45.616ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99752025-04-30T06:42:45.617ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
99762025-04-30T06:42:45.617ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
99772025-04-30T06:42:45.617ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 15 1a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56148 req_id = e0fa0640-e1a4-459f-92a1-568fbd0ff77c
99782025-04-30T06:42:45.617ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5402,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56148 req_id = e0fa0640-e1a4-459f-92a1-568fbd0ff77c
99792025-04-30T06:42:45.617ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
99802025-04-30T06:42:45.617ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99812025-04-30T06:42:45.618ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56148 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e0fa0640-e1a4-459f-92a1-568fbd0ff77c
99822025-04-30T06:42:45.618ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99832025-04-30T06:42:45.618ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
99842025-04-30T06:42:45.619ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99852025-04-30T06:42:45.619ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4230, tv_nsec: 418250669 } })
99862025-04-30T06:42:45.619ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
99872025-04-30T06:42:45.620ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c1 0c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57343 req_id = c1471929-8b86-4944-857a-d5ddbb35a3d4
99882025-04-30T06:42:45.620ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49420,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57343 req_id = c1471929-8b86-4944-857a-d5ddbb35a3d4
99892025-04-30T06:42:45.620ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99902025-04-30T06:42:45.621ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57343 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c1471929-8b86-4944-857a-d5ddbb35a3d4
99912025-04-30T06:42:45.695ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
99922025-04-30T06:42:45.696ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d9 b7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60106 req_id = ce2fdad9-ecdb-4f7a-9ed1-73b02dea0971
99932025-04-30T06:42:45.697ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55735,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60106 req_id = ce2fdad9-ecdb-4f7a-9ed1-73b02dea0971
99942025-04-30T06:42:45.697ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
99952025-04-30T06:42:45.697ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55735, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60106 req_id = ce2fdad9-ecdb-4f7a-9ed1-73b02dea0971
99962025-04-30T06:42:45.698ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
99972025-04-30T06:42:45.698ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
99982025-04-30T06:42:45.698ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
99992025-04-30T06:42:45.711ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "afe568f1-8045-430a-9ecd-f00e4eaf0a85", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100002025-04-30T06:42:45.712ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
100012025-04-30T06:42:45.715ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52d62def-b0c2-4550-a090-73360419bb18", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100022025-04-30T06:42:45.715ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
100032025-04-30T06:42:45.718ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3facb83-22a0-4464-9d55-a63382f654dd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100042025-04-30T06:42:45.718ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
100052025-04-30T06:42:45.730ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4697b03c-776c-4986-8946-68e076e87541", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100062025-04-30T06:42:45.731ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
100072025-04-30T06:42:45.738ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc469c25-0742-4779-8cc4-01b9a5f601c2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100082025-04-30T06:42:45.738ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
100092025-04-30T06:42:45.739ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
100102025-04-30T06:42:45.743ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60b60697-04d7-4c16-b0cb-c42e5c720608", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100112025-04-30T06:42:45.743ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
100122025-04-30T06:42:45.746ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1007acc-547a-485c-8044-3fa2908e9d14", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100132025-04-30T06:42:45.747ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
100142025-04-30T06:42:45.750ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d8a2395-dce8-4e6a-b0c6-a700590c601c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100152025-04-30T06:42:45.751ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
100162025-04-30T06:42:45.752ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7244b7a5-00b0-4d8e-b2e8-6b98132216f7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100172025-04-30T06:42:45.752ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
100182025-04-30T06:42:45.754ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bc2c470-cf59-47fd-a59c-63d5a836660f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100192025-04-30T06:42:45.754ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
100202025-04-30T06:42:45.755ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e504385-f3be-4b4d-a705-25b5e9fb1906", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100212025-04-30T06:42:45.755ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
100222025-04-30T06:42:45.757ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e66ace1-9b9b-4a70-a9a4-68cfc8d5a6b1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100232025-04-30T06:42:45.757ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
100242025-04-30T06:42:45.759ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43a7d5ba-b03c-400a-9a37-27cc4dd834b9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100252025-04-30T06:42:45.760ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
100262025-04-30T06:42:45.762ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5be811d1-b635-4d3a-83a9-d31fe920fde7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100272025-04-30T06:42:45.762ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
100282025-04-30T06:42:45.765ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03d6be85-478e-41fe-8d7a-c9238a6c108c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100292025-04-30T06:42:45.766ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
100302025-04-30T06:42:45.770ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddd604ba-2dd6-472d-91ca-4eaf1f8279e7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100312025-04-30T06:42:45.770ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
100322025-04-30T06:42:45.772ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4d2ad4e-110a-4a7b-b29e-6fa209658464", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100332025-04-30T06:42:45.772ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
100342025-04-30T06:42:45.775ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "995379d8-eb9b-4a22-9583-03b5e6e62dcb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100352025-04-30T06:42:45.775ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
100362025-04-30T06:42:45.777ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "633dc880-11a8-4311-871a-89759f31f334", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100372025-04-30T06:42:45.777ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
100382025-04-30T06:42:45.778ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36cf3df3-ae1f-4b2c-a580-cd5ee42a9153", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100392025-04-30T06:42:45.779ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
100402025-04-30T06:42:45.780ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3123d6d9-6163-48de-97fa-baefbcfe2fc8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100412025-04-30T06:42:45.780ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
100422025-04-30T06:42:45.782ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24f8a00c-bdf2-45f5-b49f-0177c18ef4d6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100432025-04-30T06:42:45.782ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
100442025-04-30T06:42:45.783ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b086f5d6-dbdf-41c4-bb59-eff04c191ff0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100452025-04-30T06:42:45.784ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
100462025-04-30T06:42:45.785ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3373bf68-2c60-4f9c-9d39-dba11a7415aa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100472025-04-30T06:42:45.785ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
100482025-04-30T06:42:45.787ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e504376-a88f-4452-a5a4-8af73575b7e0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100492025-04-30T06:42:45.788ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
100502025-04-30T06:42:45.789ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c4201e7-aee4-40ac-b48d-01eff333df89", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100512025-04-30T06:42:45.789ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
100522025-04-30T06:42:45.790ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100532025-04-30T06:42:45.790ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100542025-04-30T06:42:45.790ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100552025-04-30T06:42:45.791ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
100562025-04-30T06:42:45.791ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100572025-04-30T06:42:45.794ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "144bc152-5dda-4997-ae6c-a95058b2b45e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100582025-04-30T06:42:45.794ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
100592025-04-30T06:42:45.794ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100602025-04-30T06:42:45.800ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100612025-04-30T06:42:45.801ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100622025-04-30T06:42:45.801ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
100632025-04-30T06:42:45.802ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100642025-04-30T06:42:45.805ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4b6e1cb-773b-476e-9ae1-a75323e155e2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100652025-04-30T06:42:45.805ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
100662025-04-30T06:42:45.805ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100672025-04-30T06:42:45.805ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
100682025-04-30T06:42:45.806ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100692025-04-30T06:42:45.812ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecfd8f61-4a00-4e29-b8cb-70b9a31200e4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100702025-04-30T06:42:45.813ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
100712025-04-30T06:42:45.821ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0aa428a-b3bf-444c-9f59-bb3f9d4fda2e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100722025-04-30T06:42:45.821ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
100732025-04-30T06:42:45.822ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "260f6154-1a1d-4cab-b6a7-381b48aa93f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100742025-04-30T06:42:45.822ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
100752025-04-30T06:42:45.826ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9f648b-a890-48a6-8165-e1348bd587c2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100762025-04-30T06:42:45.826ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
100772025-04-30T06:42:45.829ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0560597a-ffbe-4cf0-a545-da61d5240d6a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100782025-04-30T06:42:45.829ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
100792025-04-30T06:42:45.831ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5371b0c8-d8c9-46f9-b90a-d2efeb622b92", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100802025-04-30T06:42:45.831ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
100812025-04-30T06:42:45.832ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d224f76-9293-4ec3-b08a-56dbf3a4eac9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100822025-04-30T06:42:45.832ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
100832025-04-30T06:42:45.834ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9121bbc9-2cb6-4023-9d2a-c02011f2be5b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100842025-04-30T06:42:45.834ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
100852025-04-30T06:42:45.836ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcf2c071-7b25-4c30-b749-96b46e18bc0d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100862025-04-30T06:42:45.836ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
100872025-04-30T06:42:45.839ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e4a0e0f-a4a3-47f7-8fe1-73b8580efc90", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100882025-04-30T06:42:45.839ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
100892025-04-30T06:42:45.841ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b67e269a-f18a-4599-90fe-b8c84fa43293", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100902025-04-30T06:42:45.841ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
100912025-04-30T06:42:45.854ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
100922025-04-30T06:42:45.862ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd6391d6-b2d3-48b9-801b-624f8f00dd6e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100932025-04-30T06:42:45.862ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
100942025-04-30T06:42:45.866ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f856fcb5-ae9c-4d9c-924d-facd35141372", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100952025-04-30T06:42:45.866ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
100962025-04-30T06:42:45.867ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2cbce68-acc9-4f18-bd0d-bcd0b44c02ce", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100972025-04-30T06:42:45.869ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
100982025-04-30T06:42:45.871ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbf2366c-f025-4c11-853c-254320ae7cd7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
100992025-04-30T06:42:45.871ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
101002025-04-30T06:42:45.872ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "818f4d6e-d18a-492a-b567-a996e3ba4baf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
101012025-04-30T06:42:45.872ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
101022025-04-30T06:42:45.873ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d30b3ac1-a326-4f52-ae30-852ae23e6dc2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
101032025-04-30T06:42:45.873ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101042025-04-30T06:42:45.875ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101052025-04-30T06:42:45.875ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101062025-04-30T06:42:45.875ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101072025-04-30T06:42:45.875ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101082025-04-30T06:42:45.877ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101092025-04-30T06:42:45.882ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101102025-04-30T06:42:45.882ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101112025-04-30T06:42:45.882ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101122025-04-30T06:42:45.882ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101132025-04-30T06:42:45.882ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101142025-04-30T06:42:45.882ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101152025-04-30T06:42:45.886ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101162025-04-30T06:42:45.894ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101172025-04-30T06:42:45.894ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101182025-04-30T06:42:45.894ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101192025-04-30T06:42:45.895ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101202025-04-30T06:42:45.984ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
101212025-04-30T06:42:45.985ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
101222025-04-30T06:42:45.985ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101232025-04-30T06:42:45.985ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101242025-04-30T06:42:45.985ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101252025-04-30T06:42:45.985ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
101262025-04-30T06:42:45.990ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101272025-04-30T06:42:45.991ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:42736 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
101282025-04-30T06:42:45.991ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101292025-04-30T06:42:45.991ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101302025-04-30T06:42:45.994ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
101312025-04-30T06:42:45.995ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
101322025-04-30T06:42:45.996ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:42736 req_id = c28aad8b-9ac4-4eec-bee9-bc0c516e9c83 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101332025-04-30T06:42:45.998ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 2058 local_addr = [::1]:62607 method = GET remote_addr = [::1]:42736 req_id = c28aad8b-9ac4-4eec-bee9-bc0c516e9c83 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101342025-04-30T06:42:45.998ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101352025-04-30T06:42:46.017ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101362025-04-30T06:42:46.018ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101372025-04-30T06:42:46.018ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
101382025-04-30T06:42:46.018ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101392025-04-30T06:42:46.021ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c28aad8b-9ac4-4eec-bee9-bc0c516e9c83", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:45 GMT"} })
101402025-04-30T06:42:46.024ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101412025-04-30T06:42:46.024ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101422025-04-30T06:42:46.024ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
101432025-04-30T06:42:46.030ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101442025-04-30T06:42:46.030ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101452025-04-30T06:42:46.030ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101462025-04-30T06:42:46.030ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101472025-04-30T06:42:46.031ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101482025-04-30T06:42:46.033ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
101492025-04-30T06:42:46.034ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
101502025-04-30T06:42:46.040ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101512025-04-30T06:42:46.040ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
101522025-04-30T06:42:46.041ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101532025-04-30T06:42:46.041ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
101542025-04-30T06:42:46.041ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101552025-04-30T06:42:46.043ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101562025-04-30T06:42:46.050ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101572025-04-30T06:42:46.050ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101582025-04-30T06:42:46.050ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101592025-04-30T06:42:46.051ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101602025-04-30T06:42:46.104ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
101612025-04-30T06:42:46.111ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
101622025-04-30T06:42:46.111ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
101632025-04-30T06:42:46.112ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 528.148611ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
101642025-04-30T06:42:46.112ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
101652025-04-30T06:42:46.112ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101662025-04-30T06:42:46.112ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101672025-04-30T06:42:46.112ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101682025-04-30T06:42:46.112ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101692025-04-30T06:42:46.115ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101702025-04-30T06:42:46.116ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101712025-04-30T06:42:46.128ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101722025-04-30T06:42:46.128ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101732025-04-30T06:42:46.128ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101742025-04-30T06:42:46.129ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101752025-04-30T06:42:46.145ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
101762025-04-30T06:42:46.145ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
101772025-04-30T06:42:46.146ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
101782025-04-30T06:42:46.149ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3e 23 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 >#..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49560 req_id = fd490b6b-188d-4f19-a5cb-50dc95ff23f6
101792025-04-30T06:42:46.149ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15907,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49560 req_id = fd490b6b-188d-4f19-a5cb-50dc95ff23f6
101802025-04-30T06:42:46.149ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
101812025-04-30T06:42:46.150ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101822025-04-30T06:42:46.150ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49560 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = fd490b6b-188d-4f19-a5cb-50dc95ff23f6
101832025-04-30T06:42:46.151ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4230, tv_nsec: 950656853 } })
101842025-04-30T06:42:46.152ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
101852025-04-30T06:42:46.157ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 31 19 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 1...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49767 req_id = 5609bc23-8184-49bb-a430-24875b0ec304
101862025-04-30T06:42:46.157ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12569,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49767 req_id = 5609bc23-8184-49bb-a430-24875b0ec304
101872025-04-30T06:42:46.157ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101882025-04-30T06:42:46.157ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49767 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5609bc23-8184-49bb-a430-24875b0ec304
101892025-04-30T06:42:46.225ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
101902025-04-30T06:42:46.226ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101912025-04-30T06:42:46.226ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101922025-04-30T06:42:46.227ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101932025-04-30T06:42:46.227ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
101942025-04-30T06:42:46.228ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101952025-04-30T06:42:46.229ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d7 8e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60333 req_id = 8dc4f24a-fc71-4d8d-b401-34a785d7034d
101962025-04-30T06:42:46.229ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55182,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60333 req_id = 8dc4f24a-fc71-4d8d-b401-34a785d7034d
101972025-04-30T06:42:46.229ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
101982025-04-30T06:42:46.229ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55182, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60333 req_id = 8dc4f24a-fc71-4d8d-b401-34a785d7034d
101992025-04-30T06:42:46.232ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
102002025-04-30T06:42:46.239ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
102012025-04-30T06:42:46.239ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
102022025-04-30T06:42:46.240ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102032025-04-30T06:42:46.249ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102042025-04-30T06:42:46.249ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102052025-04-30T06:42:46.249ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
102062025-04-30T06:42:46.249ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102072025-04-30T06:42:46.255ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102082025-04-30T06:42:46.255ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
102092025-04-30T06:42:46.256ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102102025-04-30T06:42:46.256ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "758df4bf-50a3-4235-8cad-78bff68a3310", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102112025-04-30T06:42:46.257ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
102122025-04-30T06:42:46.263ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c79d279d-505a-46f0-b169-6ab7f437adf4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102132025-04-30T06:42:46.263ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
102142025-04-30T06:42:46.265ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95658316-5f0b-4f1d-8cd9-a82e137e331e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102152025-04-30T06:42:46.268ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
102162025-04-30T06:42:46.272ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43ae28bd-4762-47af-bced-0bab8199eb80", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102172025-04-30T06:42:46.273ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
102182025-04-30T06:42:46.279ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "808646c6-7610-45cc-9583-7ad0c38dad17", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102192025-04-30T06:42:46.279ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
102202025-04-30T06:42:46.282ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac082250-f1b5-42dd-8763-a31fd9f25dd3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102212025-04-30T06:42:46.284ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
102222025-04-30T06:42:46.290ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6df77824-2938-4a09-a643-6f72bdeb40a9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102232025-04-30T06:42:46.290ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
102242025-04-30T06:42:46.297ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
102252025-04-30T06:42:46.299ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54d83c81-96b1-4d55-9532-1adc635e8413", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102262025-04-30T06:42:46.299ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
102272025-04-30T06:42:46.301ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50149892-61c5-4b34-b09d-0256f5d080ad", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102282025-04-30T06:42:46.302ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
102292025-04-30T06:42:46.311ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2865294-1cf8-4bbd-8953-39558e594e06", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102302025-04-30T06:42:46.312ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
102312025-04-30T06:42:46.313ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52bd3589-3b61-4237-b11c-3261946a9cd1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102322025-04-30T06:42:46.313ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
102332025-04-30T06:42:46.315ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23be1032-96f5-4ad2-b15b-e165b113e446", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102342025-04-30T06:42:46.315ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
102352025-04-30T06:42:46.317ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51c69c07-de7c-4895-8fdf-e2c01bd92891", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102362025-04-30T06:42:46.317ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
102372025-04-30T06:42:46.320ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bab658cd-1b6a-4dfc-8648-84c03af93a29", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102382025-04-30T06:42:46.320ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
102392025-04-30T06:42:46.323ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "226d4af6-0c3d-4a5f-87a2-942d868bc14c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102402025-04-30T06:42:46.323ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
102412025-04-30T06:42:46.330ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "877b9961-4a66-4e9d-b0e3-58e207ced495", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102422025-04-30T06:42:46.330ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
102432025-04-30T06:42:46.331ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e27efd1f-fc19-432d-a5ef-bddfd5bec7b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102442025-04-30T06:42:46.332ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
102452025-04-30T06:42:46.337ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "baf97db4-bd82-4398-9a90-3e17ed9fb7a8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102462025-04-30T06:42:46.337ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
102472025-04-30T06:42:46.339ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12188032-b38d-4b35-8f7f-92e6c82763dd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102482025-04-30T06:42:46.339ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
102492025-04-30T06:42:46.343ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0134a07f-0f2b-42cb-b6da-a0dc2752d4ca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102502025-04-30T06:42:46.343ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
102512025-04-30T06:42:46.344ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d86847dc-5f3d-4592-a654-0a44a26bc1cd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102522025-04-30T06:42:46.344ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
102532025-04-30T06:42:46.345ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "345be144-54fe-4835-b7ad-dbd6d8f167d2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102542025-04-30T06:42:46.345ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
102552025-04-30T06:42:46.346ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6ef6a71-33e7-49bd-81c6-03268ffb8444", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102562025-04-30T06:42:46.346ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
102572025-04-30T06:42:46.347ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f752092c-a530-4842-98da-1b89e644dd54", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102582025-04-30T06:42:46.347ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
102592025-04-30T06:42:46.349ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102602025-04-30T06:42:46.349ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102612025-04-30T06:42:46.349ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102622025-04-30T06:42:46.349ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
102632025-04-30T06:42:46.352ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102642025-04-30T06:42:46.352ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "535022ab-caa2-494c-84bb-0ba66f635fcb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102652025-04-30T06:42:46.352ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
102662025-04-30T06:42:46.362ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102672025-04-30T06:42:46.370ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102682025-04-30T06:42:46.370ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102692025-04-30T06:42:46.370ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
102702025-04-30T06:42:46.371ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102712025-04-30T06:42:46.371ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90deb459-c52c-4cbf-81e6-ff390c74c6a4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102722025-04-30T06:42:46.372ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
102732025-04-30T06:42:46.381ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102742025-04-30T06:42:46.381ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
102752025-04-30T06:42:46.388ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102762025-04-30T06:42:46.390ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df119247-f2a0-41e0-88d5-b257642eed9e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102772025-04-30T06:42:46.390ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
102782025-04-30T06:42:46.393ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a75026-d419-421f-859b-ef573c5012c0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102792025-04-30T06:42:46.396ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
102802025-04-30T06:42:46.402ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87b20d3e-f352-48ed-8bdd-ad4d05fa9262", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102812025-04-30T06:42:46.402ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
102822025-04-30T06:42:46.405ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fdf0e26-703d-40fc-91c9-77792938799b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102832025-04-30T06:42:46.405ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
102842025-04-30T06:42:46.407ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5357fdf0-b52f-412e-b10e-447fd9c1b963", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102852025-04-30T06:42:46.408ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
102862025-04-30T06:42:46.408ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78f0e796-cc78-46cd-8622-6934d9fc521a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102872025-04-30T06:42:46.409ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
102882025-04-30T06:42:46.410ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8272dbb-0b7a-481f-b202-070737349c18", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102892025-04-30T06:42:46.410ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
102902025-04-30T06:42:46.411ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4912bdf-e9c2-4712-98f4-6c09c643297f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102912025-04-30T06:42:46.411ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
102922025-04-30T06:42:46.413ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bda6cd55-0328-45ab-8c45-708f6d87d814", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102932025-04-30T06:42:46.413ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
102942025-04-30T06:42:46.417ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0f3a260-ad11-42a4-b0cd-034860784494", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102952025-04-30T06:42:46.417ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
102962025-04-30T06:42:46.424ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
102972025-04-30T06:42:46.425ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bea8305c-6875-41fb-9bed-dcbf171da2e8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
102982025-04-30T06:42:46.425ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
102992025-04-30T06:42:46.428ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6d0c59d-94ab-4ad2-ab0f-7f0776aed657", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
103002025-04-30T06:42:46.428ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
103012025-04-30T06:42:46.430ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "698159ff-1aea-43c6-beea-4c2a20893940", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
103022025-04-30T06:42:46.431ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
103032025-04-30T06:42:46.437ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da959f71-5914-4491-a220-552b04cf618b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
103042025-04-30T06:42:46.437ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
103052025-04-30T06:42:46.440ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d02f513-f13a-4b3f-8b78-ed033d7a3ef4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
103062025-04-30T06:42:46.443ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
103072025-04-30T06:42:46.453ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c3ab3ff-64e5-4f12-ab3d-8cdeb1dd8ac4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
103082025-04-30T06:42:46.454ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
103092025-04-30T06:42:46.457ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "195a7cbf-5d27-4790-be14-b34f54561713", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
103102025-04-30T06:42:46.457ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
103112025-04-30T06:42:46.459ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff26186e-b185-44bf-a115-5b01b8d74970", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
103122025-04-30T06:42:46.459ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
103132025-04-30T06:42:46.461ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ddb2ce-c10f-48d2-ac2a-42291f09e898", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
103142025-04-30T06:42:46.461ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103152025-04-30T06:42:46.462ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103162025-04-30T06:42:46.462ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103172025-04-30T06:42:46.462ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103182025-04-30T06:42:46.469ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103192025-04-30T06:42:46.476ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103202025-04-30T06:42:46.476ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103212025-04-30T06:42:46.476ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103222025-04-30T06:42:46.476ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
103232025-04-30T06:42:46.477ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103242025-04-30T06:42:46.477ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103252025-04-30T06:42:46.481ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103262025-04-30T06:42:46.481ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103272025-04-30T06:42:46.481ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103282025-04-30T06:42:46.481ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103292025-04-30T06:42:46.481ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103302025-04-30T06:42:46.481ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103312025-04-30T06:42:46.485ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103322025-04-30T06:42:46.498ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103332025-04-30T06:42:46.498ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103342025-04-30T06:42:46.498ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
103352025-04-30T06:42:46.501ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103362025-04-30T06:42:46.501ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103372025-04-30T06:42:46.508ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103382025-04-30T06:42:46.508ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103392025-04-30T06:42:46.508ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103402025-04-30T06:42:46.510ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103412025-04-30T06:42:46.519ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103422025-04-30T06:42:46.519ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
103432025-04-30T06:42:46.519ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103442025-04-30T06:42:46.653ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
103452025-04-30T06:42:46.653ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
103462025-04-30T06:42:46.656ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:37005 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
103472025-04-30T06:42:46.657ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103482025-04-30T06:42:46.657ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103492025-04-30T06:42:46.659ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:37005 req_id = a02da592-773c-41bc-b8c9-f8b4054d0ae6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103502025-04-30T06:42:46.661ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1544 local_addr = [::1]:62607 method = GET remote_addr = [::1]:37005 req_id = a02da592-773c-41bc-b8c9-f8b4054d0ae6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103512025-04-30T06:42:46.663ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103522025-04-30T06:42:46.664ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103532025-04-30T06:42:46.664ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103542025-04-30T06:42:46.664ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103552025-04-30T06:42:46.664ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a02da592-773c-41bc-b8c9-f8b4054d0ae6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
103562025-04-30T06:42:46.668ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103572025-04-30T06:42:46.669ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103582025-04-30T06:42:46.669ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
103592025-04-30T06:42:46.671ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103602025-04-30T06:42:46.671ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103612025-04-30T06:42:46.671ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103622025-04-30T06:42:46.672ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103632025-04-30T06:42:46.672ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103642025-04-30T06:42:46.676ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103652025-04-30T06:42:46.683ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103662025-04-30T06:42:46.684ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103672025-04-30T06:42:46.684ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103682025-04-30T06:42:46.684ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103692025-04-30T06:42:46.717ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
103702025-04-30T06:42:46.728ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
103712025-04-30T06:42:46.728ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
103722025-04-30T06:42:46.728ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 616.603543ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
103732025-04-30T06:42:46.728ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
103742025-04-30T06:42:46.728ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103752025-04-30T06:42:46.728ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103762025-04-30T06:42:46.728ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103772025-04-30T06:42:46.728ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103782025-04-30T06:42:46.729ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103792025-04-30T06:42:46.733ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103802025-04-30T06:42:46.737ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103812025-04-30T06:42:46.737ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103822025-04-30T06:42:46.737ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103832025-04-30T06:42:46.737ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103842025-04-30T06:42:46.742ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
103852025-04-30T06:42:46.742ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
103862025-04-30T06:42:46.743ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fc 06 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57091 req_id = caa5d910-f5ef-47e3-975d-d3533b4afec5
103872025-04-30T06:42:46.743ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64518,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57091 req_id = caa5d910-f5ef-47e3-975d-d3533b4afec5
103882025-04-30T06:42:46.743ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
103892025-04-30T06:42:46.743ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103902025-04-30T06:42:46.743ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57091 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = caa5d910-f5ef-47e3-975d-d3533b4afec5
103912025-04-30T06:42:46.744ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4231, tv_nsec: 543474433 } })
103922025-04-30T06:42:46.744ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
103932025-04-30T06:42:46.745ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7f 7a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .z..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60255 req_id = 4ff17cce-f5d3-4174-9e17-3d0028719b71
103942025-04-30T06:42:46.745ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32634,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60255 req_id = 4ff17cce-f5d3-4174-9e17-3d0028719b71
103952025-04-30T06:42:46.745ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103962025-04-30T06:42:46.745ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60255 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4ff17cce-f5d3-4174-9e17-3d0028719b71
103972025-04-30T06:42:46.819ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
103982025-04-30T06:42:46.819ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103992025-04-30T06:42:46.819ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104002025-04-30T06:42:46.820ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104012025-04-30T06:42:46.820ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
104022025-04-30T06:42:46.821ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104032025-04-30T06:42:46.822ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c9 fc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59592 req_id = 8035508c-bfee-4423-988a-8ea8adeffd54
104042025-04-30T06:42:46.822ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51708,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59592 req_id = 8035508c-bfee-4423-988a-8ea8adeffd54
104052025-04-30T06:42:46.822ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
104062025-04-30T06:42:46.823ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51708, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59592 req_id = 8035508c-bfee-4423-988a-8ea8adeffd54
104072025-04-30T06:42:46.824ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104082025-04-30T06:42:46.832ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104092025-04-30T06:42:46.832ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104102025-04-30T06:42:46.832ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
104112025-04-30T06:42:46.834ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104122025-04-30T06:42:46.835ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
104132025-04-30T06:42:46.836ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
104142025-04-30T06:42:46.836ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
104152025-04-30T06:42:46.840ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104162025-04-30T06:42:46.841ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
104172025-04-30T06:42:46.842ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104182025-04-30T06:42:46.844ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b41a3b9-440d-4c8f-a059-d9fa67cfe088", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104192025-04-30T06:42:46.844ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
104202025-04-30T06:42:46.848ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a90acd34-7160-4ab9-8f6e-3e10b7f0c382", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104212025-04-30T06:42:46.848ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
104222025-04-30T06:42:46.854ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4259f37a-9c4a-4f5e-a49c-9c5430b640f6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104232025-04-30T06:42:46.854ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
104242025-04-30T06:42:46.862ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "046b4fab-0c88-4f1e-bf5b-9c57fcdaa533", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104252025-04-30T06:42:46.863ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
104262025-04-30T06:42:46.868ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "854f2869-3aac-48d9-9c64-7bc29c0348f8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104272025-04-30T06:42:46.868ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
104282025-04-30T06:42:46.871ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91f58eb8-2762-4304-8582-75b4514d4b00", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104292025-04-30T06:42:46.871ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
104302025-04-30T06:42:46.879ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6a47c51-3c7d-423d-a472-abafd19ef090", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104312025-04-30T06:42:46.880ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
104322025-04-30T06:42:46.880ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
104332025-04-30T06:42:46.884ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "644f1a8d-1401-40eb-a3aa-cff305f9bbc7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104342025-04-30T06:42:46.884ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
104352025-04-30T06:42:46.885ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "743392a2-eea9-4b2b-8851-a96dd8884397", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104362025-04-30T06:42:46.885ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
104372025-04-30T06:42:46.887ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7a409fd-e15b-4203-9ca4-847e5d4b44ff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104382025-04-30T06:42:46.887ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
104392025-04-30T06:42:46.889ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eab35d10-2e7e-421a-9cd8-de62f9b106aa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104402025-04-30T06:42:46.890ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
104412025-04-30T06:42:46.892ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9820e87c-bee0-442c-a0ae-ae860c25a6bc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104422025-04-30T06:42:46.892ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
104432025-04-30T06:42:46.893ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d9c696f-7da6-4574-87e6-16c38799b79e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104442025-04-30T06:42:46.894ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
104452025-04-30T06:42:46.897ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1547ce39-b536-4af6-b335-bc668ac75857", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104462025-04-30T06:42:46.897ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
104472025-04-30T06:42:46.899ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "115ca082-54d1-46e6-bc1d-3951d42e5a27", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104482025-04-30T06:42:46.899ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
104492025-04-30T06:42:46.901ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
104502025-04-30T06:42:46.901ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
104512025-04-30T06:42:46.902ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
104522025-04-30T06:42:46.903ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
104532025-04-30T06:42:46.904ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3b6d9bf-f8eb-4049-82b8-7329f991f595", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104542025-04-30T06:42:46.904ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
104552025-04-30T06:42:46.907ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
104562025-04-30T06:42:46.907ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
104572025-04-30T06:42:46.908ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "664759c4-43e4-4e47-92d8-c316758b317e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104582025-04-30T06:42:46.908ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
104592025-04-30T06:42:46.911ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
104602025-04-30T06:42:46.911ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
104612025-04-30T06:42:46.912ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9914bbe-a8fb-4c6b-a1e6-3de208247c8f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104622025-04-30T06:42:46.912ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
104632025-04-30T06:42:46.914ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53092a79-5e23-4b13-b2d7-07930a2aaaad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104642025-04-30T06:42:46.914ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
104652025-04-30T06:42:46.915ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "821b1d50-7db4-4aed-90c4-02e684d0499a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104662025-04-30T06:42:46.915ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
104672025-04-30T06:42:46.916ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1f1eced-097d-40e1-8631-d9e4a0e925bf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104682025-04-30T06:42:46.917ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
104692025-04-30T06:42:46.918ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4e68f8c-13d7-4b40-bfa8-73cecf98ec6f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104702025-04-30T06:42:46.918ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
104712025-04-30T06:42:46.919ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e7c6b3d-b392-4318-b4fa-671cc8b8836c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104722025-04-30T06:42:46.919ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
104732025-04-30T06:42:46.921ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d269cbfb-ebda-46b9-b37e-9fd3a09994ec", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104742025-04-30T06:42:46.921ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
104752025-04-30T06:42:46.923ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18e0d760-f4d9-43a3-a3f7-35aba2049b12", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104762025-04-30T06:42:46.923ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
104772025-04-30T06:42:46.931ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104782025-04-30T06:42:46.931ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104792025-04-30T06:42:46.931ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104802025-04-30T06:42:46.931ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
104812025-04-30T06:42:46.932ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104822025-04-30T06:42:46.935ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7974814f-a485-4f54-bef4-dbcc62734a01", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104832025-04-30T06:42:46.936ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
104842025-04-30T06:42:46.936ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104852025-04-30T06:42:46.951ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104862025-04-30T06:42:46.951ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104872025-04-30T06:42:46.951ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
104882025-04-30T06:42:46.952ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104892025-04-30T06:42:46.956ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104902025-04-30T06:42:46.956ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
104912025-04-30T06:42:46.956ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104922025-04-30T06:42:46.956ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f16789f-5652-476a-962d-1ec08c18ae9b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104932025-04-30T06:42:46.956ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
104942025-04-30T06:42:46.960ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c4442e2-ea50-4f7a-8ea0-df2bf8aab712", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104952025-04-30T06:42:46.960ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
104962025-04-30T06:42:46.965ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25ed2c0c-68ed-41a7-b155-949f4c038c8e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104972025-04-30T06:42:46.965ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
104982025-04-30T06:42:46.974ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c8b3860-0ec7-452f-8834-cd6dcfa98fe9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
104992025-04-30T06:42:46.975ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
105002025-04-30T06:42:46.984ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0d895bc-74d8-404d-98de-d12e81926b25", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105012025-04-30T06:42:46.984ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
105022025-04-30T06:42:46.988ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e031c9d-b4bf-4299-be16-6ba27b82cdc0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105032025-04-30T06:42:46.988ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
105042025-04-30T06:42:46.991ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72e6d618-68d3-4e84-8de1-e1b5b44e0bd5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105052025-04-30T06:42:46.991ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
105062025-04-30T06:42:46.992ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e77d76d7-6906-451f-a298-edbb604f9f88", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105072025-04-30T06:42:46.993ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
105082025-04-30T06:42:46.997ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
105092025-04-30T06:42:46.998ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4426ec05-9bf1-4052-8a72-5d2af5d26ae9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105102025-04-30T06:42:46.999ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
105112025-04-30T06:42:47.003ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
105122025-04-30T06:42:47.005ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8270297a-57c8-4ca0-8ef6-db75169a1982", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105132025-04-30T06:42:47.005ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
105142025-04-30T06:42:47.009ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7da5f8c9-9c61-4d65-a05d-ae0f7e8c2232", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105152025-04-30T06:42:47.010ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
105162025-04-30T06:42:47.015ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2dc9cc8-8a96-441d-bfe0-06341f715731", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105172025-04-30T06:42:47.015ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
105182025-04-30T06:42:47.019ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c4c74e4-e3e3-4076-a4b9-12c72871ac7b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105192025-04-30T06:42:47.020ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
105202025-04-30T06:42:47.022ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d7eb0e8-077c-4c89-a11e-91851e1632ab", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105212025-04-30T06:42:47.022ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
105222025-04-30T06:42:47.024ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aca8ad95-98c8-4433-a904-d97445f01924", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105232025-04-30T06:42:47.024ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
105242025-04-30T06:42:47.025ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cef0813e-f5d4-45f3-bd39-62124f46761b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105252025-04-30T06:42:47.025ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
105262025-04-30T06:42:47.026ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "538d8d06-27fe-4e21-b426-f6f0df1060d5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105272025-04-30T06:42:47.027ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
105282025-04-30T06:42:47.028ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58be245f-1580-49b4-8923-e9764a1a47d6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105292025-04-30T06:42:47.028ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
105302025-04-30T06:42:47.029ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "468ced03-daa8-4afe-b833-4382439d74f9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105312025-04-30T06:42:47.029ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105322025-04-30T06:42:47.029ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105332025-04-30T06:42:47.029ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105342025-04-30T06:42:47.029ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105352025-04-30T06:42:47.030ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105362025-04-30T06:42:47.033ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105372025-04-30T06:42:47.039ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105382025-04-30T06:42:47.039ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105392025-04-30T06:42:47.039ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105402025-04-30T06:42:47.040ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105412025-04-30T06:42:47.040ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105422025-04-30T06:42:47.040ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105432025-04-30T06:42:47.051ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105442025-04-30T06:42:47.052ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105452025-04-30T06:42:47.052ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105462025-04-30T06:42:47.052ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
105472025-04-30T06:42:47.053ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105482025-04-30T06:42:47.053ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105492025-04-30T06:42:47.058ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105502025-04-30T06:42:47.058ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105512025-04-30T06:42:47.058ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105522025-04-30T06:42:47.059ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105532025-04-30T06:42:47.063ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105542025-04-30T06:42:47.073ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105552025-04-30T06:42:47.073ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105562025-04-30T06:42:47.073ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
105572025-04-30T06:42:47.074ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105582025-04-30T06:42:47.138ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
105592025-04-30T06:42:47.138ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
105602025-04-30T06:42:47.139ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:61050 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
105612025-04-30T06:42:47.140ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:61050 req_id = 09f0147f-cbe0-4765-8fa5-fc3eedd6fb04 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105622025-04-30T06:42:47.141ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 783 local_addr = [::1]:62607 method = GET remote_addr = [::1]:61050 req_id = 09f0147f-cbe0-4765-8fa5-fc3eedd6fb04 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105632025-04-30T06:42:47.141ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105642025-04-30T06:42:47.141ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
105652025-04-30T06:42:47.142ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105662025-04-30T06:42:47.143ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "09f0147f-cbe0-4765-8fa5-fc3eedd6fb04", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:46 GMT"} })
105672025-04-30T06:42:47.146ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105682025-04-30T06:42:47.146ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105692025-04-30T06:42:47.146ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
105702025-04-30T06:42:47.148ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105712025-04-30T06:42:47.148ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105722025-04-30T06:42:47.148ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105732025-04-30T06:42:47.149ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105742025-04-30T06:42:47.149ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105752025-04-30T06:42:47.165ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105762025-04-30T06:42:47.170ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
105772025-04-30T06:42:47.170ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105782025-04-30T06:42:47.171ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105792025-04-30T06:42:47.172ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105802025-04-30T06:42:47.222ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
105812025-04-30T06:42:47.222ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
105822025-04-30T06:42:47.222ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
105832025-04-30T06:42:47.223ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 494.296233ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
105842025-04-30T06:42:47.223ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
105852025-04-30T06:42:47.223ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105862025-04-30T06:42:47.223ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105872025-04-30T06:42:47.223ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105882025-04-30T06:42:47.223ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105892025-04-30T06:42:47.224ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105902025-04-30T06:42:47.232ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105912025-04-30T06:42:47.236ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
105922025-04-30T06:42:47.236ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105932025-04-30T06:42:47.236ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105942025-04-30T06:42:47.236ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105952025-04-30T06:42:47.243ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
105962025-04-30T06:42:47.243ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
105972025-04-30T06:42:47.245ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b8 77 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .w..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60104 req_id = 5fb27c3b-e7ac-4004-b8a7-bae8592b63b1
105982025-04-30T06:42:47.245ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47223,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60104 req_id = 5fb27c3b-e7ac-4004-b8a7-bae8592b63b1
105992025-04-30T06:42:47.245ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
106002025-04-30T06:42:47.246ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106012025-04-30T06:42:47.246ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60104 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5fb27c3b-e7ac-4004-b8a7-bae8592b63b1
106022025-04-30T06:42:47.246ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4232, tv_nsec: 45415839 } })
106032025-04-30T06:42:47.246ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
106042025-04-30T06:42:47.247ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5e d9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ^...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52873 req_id = 095d13e7-1f9a-4940-8f1b-179e40417fa9
106052025-04-30T06:42:47.247ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24281,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52873 req_id = 095d13e7-1f9a-4940-8f1b-179e40417fa9
106062025-04-30T06:42:47.247ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106072025-04-30T06:42:47.247ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52873 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 095d13e7-1f9a-4940-8f1b-179e40417fa9
106082025-04-30T06:42:47.332ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
106092025-04-30T06:42:47.332ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106102025-04-30T06:42:47.332ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106112025-04-30T06:42:47.332ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106122025-04-30T06:42:47.332ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
106132025-04-30T06:42:47.333ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106142025-04-30T06:42:47.334ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e4 25 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .%..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54736 req_id = 9947f233-f93b-4872-9c41-0286f29b42f7
106152025-04-30T06:42:47.334ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58405,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54736 req_id = 9947f233-f93b-4872-9c41-0286f29b42f7
106162025-04-30T06:42:47.334ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
106172025-04-30T06:42:47.335ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58405, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54736 req_id = 9947f233-f93b-4872-9c41-0286f29b42f7
106182025-04-30T06:42:47.336ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
106192025-04-30T06:42:47.336ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
106202025-04-30T06:42:47.336ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
106212025-04-30T06:42:47.336ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106222025-04-30T06:42:47.342ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106232025-04-30T06:42:47.342ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106242025-04-30T06:42:47.342ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
106252025-04-30T06:42:47.343ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106262025-04-30T06:42:47.347ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106272025-04-30T06:42:47.347ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
106282025-04-30T06:42:47.348ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106292025-04-30T06:42:47.348ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "66c071ad-cceb-4343-876a-9de6091574c7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106302025-04-30T06:42:47.349ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
106312025-04-30T06:42:47.353ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bb03bff-c407-4488-9915-ce312f1c48bb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106322025-04-30T06:42:47.353ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
106332025-04-30T06:42:47.361ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1853e88-1672-46ef-97ed-372d71ba53d4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106342025-04-30T06:42:47.362ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
106352025-04-30T06:42:47.364ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52e1678d-de90-4bb2-a8ee-db4b052a60df", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106362025-04-30T06:42:47.364ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
106372025-04-30T06:42:47.372ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e90d17d6-e2d3-4399-ae08-3a89255554c4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106382025-04-30T06:42:47.372ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
106392025-04-30T06:42:47.377ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cf5efab-f5c1-4919-8446-f86a23256078", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106402025-04-30T06:42:47.378ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
106412025-04-30T06:42:47.382ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b1af506-99f8-4966-8f6c-5646633baa56", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106422025-04-30T06:42:47.382ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
106432025-04-30T06:42:47.388ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
106442025-04-30T06:42:47.392ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e4bf271-ac00-4cfc-8d11-9628188d63d5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106452025-04-30T06:42:47.401ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
106462025-04-30T06:42:47.402ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d0baf42-bc0d-492d-9983-e29c09eadc3d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106472025-04-30T06:42:47.402ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
106482025-04-30T06:42:47.403ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eae7106c-87c8-432a-b9eb-aa62ba3a15ba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106492025-04-30T06:42:47.403ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
106502025-04-30T06:42:47.404ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3b29534-dcb2-44c3-95ce-b37a2106dad1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106512025-04-30T06:42:47.405ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
106522025-04-30T06:42:47.406ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41fa78bd-884e-46e8-9394-575810915843", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106532025-04-30T06:42:47.407ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
106542025-04-30T06:42:47.410ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e827d88a-f70e-426d-9a8f-0bbbe9aeedee", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106552025-04-30T06:42:47.410ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
106562025-04-30T06:42:47.412ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1126d9bb-c046-4f4d-b267-eba37a86ba98", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106572025-04-30T06:42:47.412ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
106582025-04-30T06:42:47.414ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64788f32-9363-4df0-a45d-7d2d7481365b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106592025-04-30T06:42:47.414ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
106602025-04-30T06:42:47.417ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c507de31-3208-4457-88ff-330fd81853b0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106612025-04-30T06:42:47.417ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
106622025-04-30T06:42:47.420ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "527d301e-1805-4ca6-812b-755b51026f64", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106632025-04-30T06:42:47.421ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
106642025-04-30T06:42:47.422ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14c8c3a2-e60a-4c94-9509-e272f7fced9d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106652025-04-30T06:42:47.422ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
106662025-04-30T06:42:47.427ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e239f26-3d4f-4033-9cdd-6b834289b387", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106672025-04-30T06:42:47.427ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
106682025-04-30T06:42:47.428ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac27e33c-a6d3-4b66-8793-f4bfa6667dfc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106692025-04-30T06:42:47.428ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
106702025-04-30T06:42:47.429ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f44cd05-9ec7-49d6-8463-391ec265b8d8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106712025-04-30T06:42:47.429ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
106722025-04-30T06:42:47.430ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f671a5d7-f3e7-4d49-93e9-fa457564c6cd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106732025-04-30T06:42:47.430ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
106742025-04-30T06:42:47.431ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "255a4922-62bb-4fa8-b5ce-5cf60e10364d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106752025-04-30T06:42:47.431ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
106762025-04-30T06:42:47.432ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74ba3721-f394-4c1d-a908-ae4d4ff1e9b9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106772025-04-30T06:42:47.432ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
106782025-04-30T06:42:47.434ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19c2b099-453a-497a-afc3-0d71f00294d0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106792025-04-30T06:42:47.434ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
106802025-04-30T06:42:47.436ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d6d8385-c8c9-425b-bab5-e32fe74fc6b5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106812025-04-30T06:42:47.436ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
106822025-04-30T06:42:47.438ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "816c3f3b-0c7a-41f9-a04c-8aa4cad43d63", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106832025-04-30T06:42:47.438ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
106842025-04-30T06:42:47.439ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106852025-04-30T06:42:47.439ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106862025-04-30T06:42:47.439ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106872025-04-30T06:42:47.439ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
106882025-04-30T06:42:47.440ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106892025-04-30T06:42:47.442ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106902025-04-30T06:42:47.447ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106912025-04-30T06:42:47.447ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106922025-04-30T06:42:47.447ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
106932025-04-30T06:42:47.447ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106942025-04-30T06:42:47.447ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05b6f2df-77f8-4d76-bbaa-bde014081ede", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
106952025-04-30T06:42:47.448ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
106962025-04-30T06:42:47.450ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106972025-04-30T06:42:47.450ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
106982025-04-30T06:42:47.450ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106992025-04-30T06:42:47.451ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dc06d0d-1d37-4fd5-8470-09709c3a64af", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107002025-04-30T06:42:47.451ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
107012025-04-30T06:42:47.453ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "784d643d-0635-47c5-978d-8e9bb50a796b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107022025-04-30T06:42:47.453ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
107032025-04-30T06:42:47.454ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "053802e3-ab96-4119-a0a8-4bcbdb09714d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107042025-04-30T06:42:47.454ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
107052025-04-30T06:42:47.455ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5bf9bd7-1061-4f44-9541-2ef05240d223", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107062025-04-30T06:42:47.455ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
107072025-04-30T06:42:47.457ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "726504af-844c-4d11-9c8a-4194302d8d91", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107082025-04-30T06:42:47.458ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
107092025-04-30T06:42:47.458ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8675683-7549-4c82-a6f5-6c6cd5aaceed", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107102025-04-30T06:42:47.458ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
107112025-04-30T06:42:47.459ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2105d1fe-e81f-4f23-ba61-70ca32114cb0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107122025-04-30T06:42:47.460ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
107132025-04-30T06:42:47.463ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b78d623-3202-4c61-a647-78290f72864b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107142025-04-30T06:42:47.463ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
107152025-04-30T06:42:47.466ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "140101dd-5a1e-4e36-88ef-795cf097641c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107162025-04-30T06:42:47.466ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
107172025-04-30T06:42:47.469ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28cfa002-67b6-43a7-a3c9-fd48f1a4f0e6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107182025-04-30T06:42:47.469ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
107192025-04-30T06:42:47.471ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df0590ae-9e3a-461c-aade-af41ae8597c9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107202025-04-30T06:42:47.471ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
107212025-04-30T06:42:47.473ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f25235a7-6e5a-4adf-a77a-1bb69344c16c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107222025-04-30T06:42:47.473ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
107232025-04-30T06:42:47.475ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "992881c7-8497-4dba-9f57-d7f5a7f18390", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107242025-04-30T06:42:47.475ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
107252025-04-30T06:42:47.476ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "658dd472-5f9d-40ba-a02b-ea2c9fa138de", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107262025-04-30T06:42:47.476ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
107272025-04-30T06:42:47.483ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b048b2c-df58-45c2-9760-9db7f60503a4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107282025-04-30T06:42:47.483ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
107292025-04-30T06:42:47.489ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
107302025-04-30T06:42:47.499ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
107312025-04-30T06:42:47.499ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107322025-04-30T06:42:47.500ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = f9bc0244-c8fb-4602-8005-3621271d8f55 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107332025-04-30T06:42:47.500ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = f9bc0244-c8fb-4602-8005-3621271d8f55 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107342025-04-30T06:42:47.500ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = f9bc0244-c8fb-4602-8005-3621271d8f55 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107352025-04-30T06:42:47.501ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = f9bc0244-c8fb-4602-8005-3621271d8f55 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107362025-04-30T06:42:47.501ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5414942-941f-4e1b-9208-238791599a27", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107372025-04-30T06:42:47.502ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
107382025-04-30T06:42:47.503ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "faa49c3e-fb23-478f-8159-cd1ba1c89c27", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107392025-04-30T06:42:47.503ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107402025-04-30T06:42:47.503ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107412025-04-30T06:42:47.503ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107422025-04-30T06:42:47.503ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107432025-04-30T06:42:47.504ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107442025-04-30T06:42:47.504ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 4541 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = f9bc0244-c8fb-4602-8005-3621271d8f55 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
107452025-04-30T06:42:47.505ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9bc0244-c8fb-4602-8005-3621271d8f55", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107462025-04-30T06:42:47.505ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
107472025-04-30T06:42:47.506ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3fc68b50-aefa-4be1-a183-9e2dc0adbadf uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
107482025-04-30T06:42:47.506ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3fc68b50-aefa-4be1-a183-9e2dc0adbadf resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
107492025-04-30T06:42:47.507ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3fc68b50-aefa-4be1-a183-9e2dc0adbadf roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
107502025-04-30T06:42:47.508ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3fc68b50-aefa-4be1-a183-9e2dc0adbadf resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
107512025-04-30T06:42:47.508ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107522025-04-30T06:42:47.512ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107532025-04-30T06:42:47.513ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107542025-04-30T06:42:47.513ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107552025-04-30T06:42:47.513ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107562025-04-30T06:42:47.513ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107572025-04-30T06:42:47.513ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107582025-04-30T06:42:47.517ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 10652 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3fc68b50-aefa-4be1-a183-9e2dc0adbadf response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
107592025-04-30T06:42:47.519ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107602025-04-30T06:42:47.523ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107612025-04-30T06:42:47.525ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107622025-04-30T06:42:47.525ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107632025-04-30T06:42:47.526ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107642025-04-30T06:42:47.528ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc68b50-aefa-4be1-a183-9e2dc0adbadf", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107652025-04-30T06:42:47.528ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107662025-04-30T06:42:47.528ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
107672025-04-30T06:42:47.528ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
107682025-04-30T06:42:47.565ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107692025-04-30T06:42:47.565ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107702025-04-30T06:42:47.565ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107712025-04-30T06:42:47.565ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
107722025-04-30T06:42:47.566ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107732025-04-30T06:42:47.567ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107742025-04-30T06:42:47.567ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107752025-04-30T06:42:47.649ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
107762025-04-30T06:42:47.649ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
107772025-04-30T06:42:47.649ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:62880 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
107782025-04-30T06:42:47.652ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107792025-04-30T06:42:47.657ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107802025-04-30T06:42:47.657ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107812025-04-30T06:42:47.657ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
107822025-04-30T06:42:47.657ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107832025-04-30T06:42:47.657ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:62880 req_id = 6c7590d2-3abf-43fd-8570-f4277e7927c1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
107842025-04-30T06:42:47.658ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107852025-04-30T06:42:47.658ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107862025-04-30T06:42:47.658ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107872025-04-30T06:42:47.658ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107882025-04-30T06:42:47.659ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1762 local_addr = [::1]:62607 method = GET remote_addr = [::1]:62880 req_id = 6c7590d2-3abf-43fd-8570-f4277e7927c1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
107892025-04-30T06:42:47.661ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c7590d2-3abf-43fd-8570-f4277e7927c1", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
107902025-04-30T06:42:47.663ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
107912025-04-30T06:42:47.663ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
107922025-04-30T06:42:47.663ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
107932025-04-30T06:42:47.665ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
107942025-04-30T06:42:47.665ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107952025-04-30T06:42:47.666ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107962025-04-30T06:42:47.666ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107972025-04-30T06:42:47.666ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107982025-04-30T06:42:47.667ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107992025-04-30T06:42:47.667ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
108002025-04-30T06:42:47.668ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108012025-04-30T06:42:47.670ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108022025-04-30T06:42:47.697ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108032025-04-30T06:42:47.697ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108042025-04-30T06:42:47.697ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108052025-04-30T06:42:47.698ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108062025-04-30T06:42:47.742ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
108072025-04-30T06:42:47.744ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
108082025-04-30T06:42:47.745ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
108092025-04-30T06:42:47.746ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 523.021827ms iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
108102025-04-30T06:42:47.746ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
108112025-04-30T06:42:47.746ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108122025-04-30T06:42:47.746ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108132025-04-30T06:42:47.746ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108142025-04-30T06:42:47.746ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108152025-04-30T06:42:47.748ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108162025-04-30T06:42:47.751ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108172025-04-30T06:42:47.756ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108182025-04-30T06:42:47.756ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108192025-04-30T06:42:47.756ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108202025-04-30T06:42:47.756ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108212025-04-30T06:42:47.760ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
108222025-04-30T06:42:47.760ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
108232025-04-30T06:42:47.760ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e4 47 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .G..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50690 req_id = 6241c2bf-5d5a-4d1d-a5d3-26d82837eb86
108242025-04-30T06:42:47.761ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58439,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50690 req_id = 6241c2bf-5d5a-4d1d-a5d3-26d82837eb86
108252025-04-30T06:42:47.761ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
108262025-04-30T06:42:47.761ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108272025-04-30T06:42:47.761ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50690 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6241c2bf-5d5a-4d1d-a5d3-26d82837eb86
108282025-04-30T06:42:47.762ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4232, tv_nsec: 560721071 } })
108292025-04-30T06:42:47.762ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
108302025-04-30T06:42:47.763ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c3 29 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .)..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52091 req_id = d8bb227c-ab81-4db1-8e5e-bd3b508e0259
108312025-04-30T06:42:47.763ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49961,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52091 req_id = d8bb227c-ab81-4db1-8e5e-bd3b508e0259
108322025-04-30T06:42:47.763ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108332025-04-30T06:42:47.764ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52091 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d8bb227c-ab81-4db1-8e5e-bd3b508e0259
108342025-04-30T06:42:47.834ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
108352025-04-30T06:42:47.834ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108362025-04-30T06:42:47.834ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108372025-04-30T06:42:47.834ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108382025-04-30T06:42:47.834ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
108392025-04-30T06:42:47.835ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108402025-04-30T06:42:47.836ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2f 53 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c /S..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53597 req_id = b34b4354-898d-4892-bb23-d2bf3f96a492
108412025-04-30T06:42:47.836ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12115,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53597 req_id = b34b4354-898d-4892-bb23-d2bf3f96a492
108422025-04-30T06:42:47.836ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
108432025-04-30T06:42:47.836ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12115, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53597 req_id = b34b4354-898d-4892-bb23-d2bf3f96a492
108442025-04-30T06:42:47.836ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
108452025-04-30T06:42:47.836ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
108462025-04-30T06:42:47.836ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
108472025-04-30T06:42:47.838ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108482025-04-30T06:42:47.846ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108492025-04-30T06:42:47.846ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108502025-04-30T06:42:47.846ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
108512025-04-30T06:42:47.847ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108522025-04-30T06:42:47.847ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "70e1e1c8-d33d-4cde-aa20-a1373b14fc57", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
108532025-04-30T06:42:47.847ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
108542025-04-30T06:42:47.854ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
108552025-04-30T06:42:47.854ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
108562025-04-30T06:42:47.854ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
108572025-04-30T06:42:47.855ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
108582025-04-30T06:42:47.855ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
108592025-04-30T06:42:47.855ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
108602025-04-30T06:42:47.855ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
108612025-04-30T06:42:47.856ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
108622025-04-30T06:42:47.856ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
108632025-04-30T06:42:47.856ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108642025-04-30T06:42:47.856ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
108652025-04-30T06:42:47.857ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108662025-04-30T06:42:47.857ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
108672025-04-30T06:42:47.857ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 35.471\u{b5}s iteration = 3 status = {"error":"task disabled"}
108682025-04-30T06:42:47.858ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
108692025-04-30T06:42:47.858ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
108702025-04-30T06:42:47.858ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108712025-04-30T06:42:47.858ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
108722025-04-30T06:42:47.858ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108732025-04-30T06:42:47.860ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108742025-04-30T06:42:47.860ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
108752025-04-30T06:42:47.860ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
108762025-04-30T06:42:47.860ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108772025-04-30T06:42:47.860ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
108782025-04-30T06:42:47.860ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
108792025-04-30T06:42:47.860ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
108802025-04-30T06:42:47.870ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
108812025-04-30T06:42:47.870ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
108822025-04-30T06:42:47.873ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
108832025-04-30T06:42:47.873ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
108842025-04-30T06:42:47.873ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
108852025-04-30T06:42:47.873ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
108862025-04-30T06:42:47.873ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
108872025-04-30T06:42:47.873ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
108882025-04-30T06:42:47.873ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108892025-04-30T06:42:47.873ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
108902025-04-30T06:42:47.873ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
108912025-04-30T06:42:47.874ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
108922025-04-30T06:42:47.874ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
108932025-04-30T06:42:47.874ZINFOtest_delete_snapshot (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
108942025-04-30T06:42:47.874ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = instance_updater elapsed = 77.284\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
108952025-04-30T06:42:47.874ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
108962025-04-30T06:42:47.874ZINFOtest_delete_snapshot (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
108972025-04-30T06:42:47.874ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
108982025-04-30T06:42:47.875ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108992025-04-30T06:42:47.875ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109002025-04-30T06:42:47.875ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109012025-04-30T06:42:47.875ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109022025-04-30T06:42:47.878ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 21.179499ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
109032025-04-30T06:42:47.879ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ef a1 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61396 req_id = 03543b96-6c9e-4859-8f56-a25cbcc1b0d0
109042025-04-30T06:42:47.880ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61345,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61396 req_id = 03543b96-6c9e-4859-8f56-a25cbcc1b0d0
109052025-04-30T06:42:47.880ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
109062025-04-30T06:42:47.880ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109072025-04-30T06:42:47.880ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61396 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 03543b96-6c9e-4859-8f56-a25cbcc1b0d0
109082025-04-30T06:42:47.880ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 26 35 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 &5..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63977 req_id = 0b97253d-f53d-4224-baec-16f237d249a7
109092025-04-30T06:42:47.880ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9781,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63977 req_id = 0b97253d-f53d-4224-baec-16f237d249a7
109102025-04-30T06:42:47.880ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
109112025-04-30T06:42:47.881ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109122025-04-30T06:42:47.881ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63977 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0b97253d-f53d-4224-baec-16f237d249a7
109132025-04-30T06:42:47.884ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4232, tv_nsec: 682336912 } })
109142025-04-30T06:42:47.884ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
109152025-04-30T06:42:47.884ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4232, tv_nsec: 682692050 } })
109162025-04-30T06:42:47.884ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
109172025-04-30T06:42:47.884ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66207b3f-934f-4699-a62c-1a8240583e3a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
109182025-04-30T06:42:47.884ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
109192025-04-30T06:42:47.885ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 51 69 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Qi..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63270 req_id = 4e8b77ec-ed66-4c27-b3e7-ae5b4fa6c92e
109202025-04-30T06:42:47.885ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20841,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63270 req_id = 4e8b77ec-ed66-4c27-b3e7-ae5b4fa6c92e
109212025-04-30T06:42:47.885ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109222025-04-30T06:42:47.885ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63270 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4e8b77ec-ed66-4c27-b3e7-ae5b4fa6c92e
109232025-04-30T06:42:47.885ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ab c9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60953 req_id = 2e3b662f-caf9-4cfe-b09d-31f64bc666fc
109242025-04-30T06:42:47.886ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43977,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60953 req_id = 2e3b662f-caf9-4cfe-b09d-31f64bc666fc
109252025-04-30T06:42:47.886ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109262025-04-30T06:42:47.886ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60953 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2e3b662f-caf9-4cfe-b09d-31f64bc666fc
109272025-04-30T06:42:47.886ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
109282025-04-30T06:42:47.951ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
109292025-04-30T06:42:47.951ZDEBGtest_delete_snapshot (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
109302025-04-30T06:42:47.952ZDEBGtest_delete_snapshot (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
109312025-04-30T06:42:47.952ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
109322025-04-30T06:42:47.956ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
109332025-04-30T06:42:47.958ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
109342025-04-30T06:42:48.022ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
109352025-04-30T06:42:48.022ZDEBGtest_delete_snapshot (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
109362025-04-30T06:42:48.023ZDEBGtest_delete_snapshot (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
109372025-04-30T06:42:48.023ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
109382025-04-30T06:42:48.025ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
109392025-04-30T06:42:48.025ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109402025-04-30T06:42:48.030ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
109412025-04-30T06:42:48.030ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
109422025-04-30T06:42:48.030ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
109432025-04-30T06:42:48.030ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
109442025-04-30T06:42:48.041ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109452025-04-30T06:42:48.041ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109462025-04-30T06:42:48.041ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
109472025-04-30T06:42:48.043ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
109482025-04-30T06:42:48.043ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109492025-04-30T06:42:48.048ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
109502025-04-30T06:42:48.049ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109512025-04-30T06:42:48.049ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109522025-04-30T06:42:48.049ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109532025-04-30T06:42:48.052ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109542025-04-30T06:42:48.057ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
109552025-04-30T06:42:48.057ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
109562025-04-30T06:42:48.057ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
109572025-04-30T06:42:48.058ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
109582025-04-30T06:42:48.068ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109592025-04-30T06:42:48.075ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109602025-04-30T06:42:48.076ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
109612025-04-30T06:42:48.077ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109622025-04-30T06:42:48.077ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109632025-04-30T06:42:48.077ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109642025-04-30T06:42:48.077ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109652025-04-30T06:42:48.078ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
109662025-04-30T06:42:48.080ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109672025-04-30T06:42:48.080ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109682025-04-30T06:42:48.082ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109692025-04-30T06:42:48.093ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
109702025-04-30T06:42:48.093ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
109712025-04-30T06:42:48.099ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: b2 05 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62445 req_id = edb92b48-848b-47db-8ace-4db995e5dc91
109722025-04-30T06:42:48.099ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45573,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62445 req_id = edb92b48-848b-47db-8ace-4db995e5dc91
109732025-04-30T06:42:48.099ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
109742025-04-30T06:42:48.100ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109752025-04-30T06:42:48.100ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62445 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = edb92b48-848b-47db-8ace-4db995e5dc91
109762025-04-30T06:42:48.101ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109772025-04-30T06:42:48.108ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109782025-04-30T06:42:48.108ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109792025-04-30T06:42:48.108ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109802025-04-30T06:42:48.109ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109812025-04-30T06:42:48.111ZINFOtest_delete_snapshot (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
109822025-04-30T06:42:48.111ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 237.84425ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
109832025-04-30T06:42:48.115ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109842025-04-30T06:42:48.115ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109852025-04-30T06:42:48.115ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109862025-04-30T06:42:48.118ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
109872025-04-30T06:42:48.118ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109882025-04-30T06:42:48.118ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4232, tv_nsec: 917367215 } })
109892025-04-30T06:42:48.118ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
109902025-04-30T06:42:48.128ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 22 d5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e "...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63338 req_id = eeec6bbf-d583-4998-9176-4737e1ea4e79
109912025-04-30T06:42:48.129ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8917,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63338 req_id = eeec6bbf-d583-4998-9176-4737e1ea4e79
109922025-04-30T06:42:48.129ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109932025-04-30T06:42:48.129ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63338 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = eeec6bbf-d583-4998-9176-4737e1ea4e79
109942025-04-30T06:42:48.132ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
109952025-04-30T06:42:48.207ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
109962025-04-30T06:42:48.208ZDEBGtest_delete_snapshot (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
109972025-04-30T06:42:48.208ZDEBGtest_delete_snapshot (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
109982025-04-30T06:42:48.208ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
109992025-04-30T06:42:48.210ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
110002025-04-30T06:42:48.288ZINFOtest_delete_snapshot (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
110012025-04-30T06:42:48.288ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:62607/vpc-routes
110022025-04-30T06:42:48.289ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
110032025-04-30T06:42:48.289ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110042025-04-30T06:42:48.289ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110052025-04-30T06:42:48.289ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110062025-04-30T06:42:48.291ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110072025-04-30T06:42:48.294ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110082025-04-30T06:42:48.299ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
110092025-04-30T06:42:48.299ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
110102025-04-30T06:42:48.299ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110112025-04-30T06:42:48.299ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110122025-04-30T06:42:48.299ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110132025-04-30T06:42:48.300ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110142025-04-30T06:42:48.302ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:32940 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
110152025-04-30T06:42:48.303ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
110162025-04-30T06:42:48.306ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:32940 req_id = 284c2503-f74c-4a4b-ba1d-37cbd6e99a71 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
110172025-04-30T06:42:48.311ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 4970 local_addr = [::1]:62607 method = GET remote_addr = [::1]:32940 req_id = 284c2503-f74c-4a4b-ba1d-37cbd6e99a71 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
110182025-04-30T06:42:48.312ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110192025-04-30T06:42:48.319ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
110202025-04-30T06:42:48.319ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110212025-04-30T06:42:48.319ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110222025-04-30T06:42:48.321ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110232025-04-30T06:42:48.335ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:62607/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "284c2503-f74c-4a4b-ba1d-37cbd6e99a71", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:47 GMT"} })
110242025-04-30T06:42:48.335ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110252025-04-30T06:42:48.335ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110262025-04-30T06:42:48.336ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110272025-04-30T06:42:48.402ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:62607/v2p
110282025-04-30T06:42:48.405ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:39966 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
110292025-04-30T06:42:48.415ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110302025-04-30T06:42:48.415ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110312025-04-30T06:42:48.416ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110322025-04-30T06:42:48.417ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:39966 req_id = bb21f308-d700-46e8-89b6-cb4e9d8aa9c7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
110332025-04-30T06:42:48.419ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 2487 local_addr = [::1]:62607 method = GET remote_addr = [::1]:39966 req_id = bb21f308-d700-46e8-89b6-cb4e9d8aa9c7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
110342025-04-30T06:42:48.420ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
110352025-04-30T06:42:48.421ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:62607/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb21f308-d700-46e8-89b6-cb4e9d8aa9c7", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110362025-04-30T06:42:48.421ZINFOtest_delete_snapshot (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
110372025-04-30T06:42:48.421ZINFOtest_delete_snapshot (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
110382025-04-30T06:42:48.421ZINFOtest_delete_snapshot (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
110392025-04-30T06:42:48.424ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 551.756118ms iteration = 2 status = {}
110402025-04-30T06:42:48.425ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
110412025-04-30T06:42:48.425ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110422025-04-30T06:42:48.426ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110432025-04-30T06:42:48.426ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110442025-04-30T06:42:48.426ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110452025-04-30T06:42:48.450ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77bba0db-b0cc-4afc-a7a6-0ff9061519f2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110462025-04-30T06:42:48.450ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
110472025-04-30T06:42:48.455ZINFOtest_delete_snapshot (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
110482025-04-30T06:42:48.455ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:62607/eip-gateways
110492025-04-30T06:42:48.460ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = PUT remote_addr = [::1]:32940 req_id = afdaad15-7982-4b1e-bbb3-64b3ec6d8c82 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
110502025-04-30T06:42:48.461ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1291 local_addr = [::1]:62607 method = PUT remote_addr = [::1]:32940 req_id = afdaad15-7982-4b1e-bbb3-64b3ec6d8c82 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
110512025-04-30T06:42:48.462ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110522025-04-30T06:42:48.467ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
110532025-04-30T06:42:48.467ZERROtest_delete_snapshot (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
110542025-04-30T06:42:48.467ZERROtest_delete_snapshot (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
110552025-04-30T06:42:48.467ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 608.430639ms iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
110562025-04-30T06:42:48.469ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:62607/eip-gateways", status: 204, headers: {"x-request-id": "afdaad15-7982-4b1e-bbb3-64b3ec6d8c82", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110572025-04-30T06:42:48.472ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 597.449259ms iteration = 2 status = {}
110582025-04-30T06:42:48.473ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110592025-04-30T06:42:48.473ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110602025-04-30T06:42:48.473ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110612025-04-30T06:42:48.473ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
110622025-04-30T06:42:48.473ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110632025-04-30T06:42:48.482ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110642025-04-30T06:42:48.500ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
110652025-04-30T06:42:48.501ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110662025-04-30T06:42:48.501ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
110672025-04-30T06:42:48.506ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110682025-04-30T06:42:48.510ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf7ea1d8-67f8-4b99-b638-4dfd593442b9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110692025-04-30T06:42:48.511ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
110702025-04-30T06:42:48.511ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110712025-04-30T06:42:48.511ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
110722025-04-30T06:42:48.512ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110732025-04-30T06:42:48.518ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4928d2f6-100a-4741-a87d-c433d2a3c049", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110742025-04-30T06:42:48.518ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
110752025-04-30T06:42:48.527ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43d7af16-31a8-454e-8fd4-08b8cc55c85b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110762025-04-30T06:42:48.527ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
110772025-04-30T06:42:48.530ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d082aaa-0d4d-4176-b8bc-85bc33824609", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110782025-04-30T06:42:48.531ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
110792025-04-30T06:42:48.535ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f589c8e-cad0-48f6-a807-25dd69d57869", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110802025-04-30T06:42:48.536ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
110812025-04-30T06:42:48.550ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0b67fef-1d2c-4dad-99fc-e6444ad436e5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110822025-04-30T06:42:48.551ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
110832025-04-30T06:42:48.553ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110842025-04-30T06:42:48.553ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110852025-04-30T06:42:48.563ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
110862025-04-30T06:42:48.564ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9424645f-ee5a-4de3-b2c0-cb0e0ebf2eec", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110872025-04-30T06:42:48.564ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
110882025-04-30T06:42:48.564ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110892025-04-30T06:42:48.566ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110902025-04-30T06:42:48.566ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110912025-04-30T06:42:48.566ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110922025-04-30T06:42:48.569ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a79118a1-5c50-494a-9f37-37b694df9d53", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110932025-04-30T06:42:48.569ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
110942025-04-30T06:42:48.571ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8384534-44fc-4add-b1ff-3390ed2dffe4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110952025-04-30T06:42:48.571ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
110962025-04-30T06:42:48.574ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb08dab3-8bf0-46db-bb69-3ca8a2c44b6e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110972025-04-30T06:42:48.574ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
110982025-04-30T06:42:48.583ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84b8490e-9974-4a06-a46f-5b405212d02d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
110992025-04-30T06:42:48.583ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
111002025-04-30T06:42:48.587ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a775c5c-68a2-4040-837c-615485e84d9e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111012025-04-30T06:42:48.587ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
111022025-04-30T06:42:48.592ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d953b6f8-9c3f-4a7e-a8c0-50b5e6f86b39", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111032025-04-30T06:42:48.593ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
111042025-04-30T06:42:48.595ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b26107bd-1a51-4d1d-9206-ea51d25793b8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111052025-04-30T06:42:48.595ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
111062025-04-30T06:42:48.597ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97325e89-a45e-437c-a8c7-d1f8f2aeac69", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111072025-04-30T06:42:48.598ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
111082025-04-30T06:42:48.601ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f352b80-f434-4eac-88ae-121e86da91a4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111092025-04-30T06:42:48.601ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
111102025-04-30T06:42:48.602ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b18444c-fc63-4cf0-9a7b-909d998f33e4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111112025-04-30T06:42:48.602ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
111122025-04-30T06:42:48.608ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4db1777-6061-4a6e-ab99-9fc9e74dc33f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111132025-04-30T06:42:48.608ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
111142025-04-30T06:42:48.611ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e02fa399-e803-46b1-b4fd-4257b5df7431", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111152025-04-30T06:42:48.611ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
111162025-04-30T06:42:48.613ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93c3f802-fb08-464b-98b7-af0d7c04440b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111172025-04-30T06:42:48.613ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
111182025-04-30T06:42:48.613ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f17cc39-b40c-46fe-9629-1e0b88599f5a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111192025-04-30T06:42:48.614ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
111202025-04-30T06:42:48.614ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111212025-04-30T06:42:48.614ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111222025-04-30T06:42:48.614ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111232025-04-30T06:42:48.614ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
111242025-04-30T06:42:48.615ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111252025-04-30T06:42:48.625ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111262025-04-30T06:42:48.631ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111272025-04-30T06:42:48.631ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111282025-04-30T06:42:48.632ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
111292025-04-30T06:42:48.633ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111302025-04-30T06:42:48.636ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111312025-04-30T06:42:48.636ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
111322025-04-30T06:42:48.637ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111332025-04-30T06:42:48.639ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8774ba81-7445-4e3b-8a38-83c5409d3219", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111342025-04-30T06:42:48.639ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
111352025-04-30T06:42:48.669ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4585ebce-0de2-4ce4-90a8-cbdd95c28625", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111362025-04-30T06:42:48.670ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
111372025-04-30T06:42:48.707ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5850cc50-154e-45b0-a1b2-469d7ded6200", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111382025-04-30T06:42:48.707ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
111392025-04-30T06:42:48.710ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5256dc1f-36d2-4157-8178-c77271d7aa0a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111402025-04-30T06:42:48.710ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
111412025-04-30T06:42:48.722ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b368a63-aca2-4757-bcbd-21098550af78", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111422025-04-30T06:42:48.722ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
111432025-04-30T06:42:48.725ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cacd3485-a223-499a-accd-5433d0c36ce5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111442025-04-30T06:42:48.725ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
111452025-04-30T06:42:48.726ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6486cdcc-cb2e-473c-b7f3-68d25b2043b3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111462025-04-30T06:42:48.726ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
111472025-04-30T06:42:48.728ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31933717-2c41-4f1f-85a8-4cdd6cdd6c8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111482025-04-30T06:42:48.728ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
111492025-04-30T06:42:48.742ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
111502025-04-30T06:42:48.747ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "711b83c2-eb79-483a-97b9-e4c9851ec6f7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111512025-04-30T06:42:48.747ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
111522025-04-30T06:42:48.751ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf2a2cac-8449-485a-981e-d6d4e4289c8f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111532025-04-30T06:42:48.751ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
111542025-04-30T06:42:48.753ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83be6f3c-e6bc-43ba-a8aa-a61bc59bb3c1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111552025-04-30T06:42:48.753ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
111562025-04-30T06:42:48.755ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e72df215-cfd2-41e4-9c20-d15ddacec975", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111572025-04-30T06:42:48.755ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
111582025-04-30T06:42:48.765ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "170ec3b9-1209-4db8-98b3-47e5e81b0808", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111592025-04-30T06:42:48.765ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
111602025-04-30T06:42:48.774ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab92c351-a95d-4cd5-bf48-1f631ec2f6a5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111612025-04-30T06:42:48.774ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
111622025-04-30T06:42:48.793ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6044286-cae1-4254-aa7d-84129816182c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111632025-04-30T06:42:48.793ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
111642025-04-30T06:42:48.796ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63876cb3-ba1e-4386-bebf-fe33404b2807", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111652025-04-30T06:42:48.796ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
111662025-04-30T06:42:48.798ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111672025-04-30T06:42:48.798ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111682025-04-30T06:42:48.798ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111692025-04-30T06:42:48.798ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
111702025-04-30T06:42:48.801ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111712025-04-30T06:42:48.804ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ecb4b91-d717-478c-b9ac-65a4f1904749", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111722025-04-30T06:42:48.804ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
111732025-04-30T06:42:48.805ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111742025-04-30T06:42:48.818ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111752025-04-30T06:42:48.818ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111762025-04-30T06:42:48.818ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
111772025-04-30T06:42:48.819ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111782025-04-30T06:42:48.824ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83b7df7b-c48e-4c64-9069-b11011739332", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111792025-04-30T06:42:48.825ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
111802025-04-30T06:42:48.826ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111812025-04-30T06:42:48.826ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
111822025-04-30T06:42:48.827ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111832025-04-30T06:42:48.861ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62d616e0-219b-4e89-89a0-b886ce2e2a6d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111842025-04-30T06:42:48.861ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
111852025-04-30T06:42:48.864ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "679fd511-df64-4d86-9cf4-d2636b92b94d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111862025-04-30T06:42:48.864ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
111872025-04-30T06:42:48.865ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53698120-cedb-4178-859b-98fa25786c91", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
111882025-04-30T06:42:48.865ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
111892025-04-30T06:42:48.865ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111902025-04-30T06:42:48.866ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111912025-04-30T06:42:48.866ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111922025-04-30T06:42:48.889ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111932025-04-30T06:42:48.893ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111942025-04-30T06:42:48.898ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
111952025-04-30T06:42:48.898ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
111962025-04-30T06:42:48.898ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111972025-04-30T06:42:48.898ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111982025-04-30T06:42:48.898ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111992025-04-30T06:42:48.899ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112002025-04-30T06:42:48.901ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
112012025-04-30T06:42:48.901ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
112022025-04-30T06:42:48.901ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
112032025-04-30T06:42:48.903ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
112042025-04-30T06:42:48.908ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112052025-04-30T06:42:48.915ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112062025-04-30T06:42:48.915ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112072025-04-30T06:42:48.915ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112082025-04-30T06:42:48.916ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112092025-04-30T06:42:48.917ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
112102025-04-30T06:42:48.917ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
112112025-04-30T06:42:48.922ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
112122025-04-30T06:42:48.922ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
112132025-04-30T06:42:48.983ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
112142025-04-30T06:42:48.984ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
112152025-04-30T06:42:48.985ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:39315 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
112162025-04-30T06:42:48.985ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:39315 req_id = 222cf8b6-a455-4d1e-bf0c-159b5802951f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
112172025-04-30T06:42:48.986ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 637 local_addr = [::1]:62607 method = GET remote_addr = [::1]:39315 req_id = 222cf8b6-a455-4d1e-bf0c-159b5802951f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
112182025-04-30T06:42:48.987ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "222cf8b6-a455-4d1e-bf0c-159b5802951f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
112192025-04-30T06:42:48.990ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
112202025-04-30T06:42:48.990ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
112212025-04-30T06:42:48.990ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
112222025-04-30T06:42:48.993ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
112232025-04-30T06:42:48.993ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112242025-04-30T06:42:48.993ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112252025-04-30T06:42:48.993ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112262025-04-30T06:42:48.994ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112272025-04-30T06:42:48.997ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112282025-04-30T06:42:49.007ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112292025-04-30T06:42:49.007ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112302025-04-30T06:42:49.007ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112312025-04-30T06:42:49.008ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112322025-04-30T06:42:49.009ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
112332025-04-30T06:42:49.017ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
112342025-04-30T06:42:49.054ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
112352025-04-30T06:42:49.054ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
112362025-04-30T06:42:49.055ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.308973394s iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
112372025-04-30T06:42:49.055ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
112382025-04-30T06:42:49.055ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
112392025-04-30T06:42:49.055ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112402025-04-30T06:42:49.055ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112412025-04-30T06:42:49.055ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112422025-04-30T06:42:49.057ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112432025-04-30T06:42:49.062ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112442025-04-30T06:42:49.072ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112452025-04-30T06:42:49.073ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112462025-04-30T06:42:49.073ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112472025-04-30T06:42:49.073ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112482025-04-30T06:42:49.074ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112492025-04-30T06:42:49.074ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112502025-04-30T06:42:49.074ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112512025-04-30T06:42:49.074ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
112522025-04-30T06:42:49.074ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112532025-04-30T06:42:49.078ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
112542025-04-30T06:42:49.078ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
112552025-04-30T06:42:49.080ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112562025-04-30T06:42:49.085ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112572025-04-30T06:42:49.085ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112582025-04-30T06:42:49.085ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
112592025-04-30T06:42:49.085ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112602025-04-30T06:42:49.086ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: df e3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52562 req_id = 4c89cd10-7d7f-41ec-995b-434c53b371b5
112612025-04-30T06:42:49.086ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57315,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52562 req_id = 4c89cd10-7d7f-41ec-995b-434c53b371b5
112622025-04-30T06:42:49.086ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
112632025-04-30T06:42:49.086ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112642025-04-30T06:42:49.086ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52562 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4c89cd10-7d7f-41ec-995b-434c53b371b5
112652025-04-30T06:42:49.087ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4233, tv_nsec: 886325781 } })
112662025-04-30T06:42:49.087ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
112672025-04-30T06:42:49.093ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 40 21 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e @!..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59776 req_id = 12e5a7a8-d63d-409f-91e2-6cad56c9b996
112682025-04-30T06:42:49.093ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16417,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59776 req_id = 12e5a7a8-d63d-409f-91e2-6cad56c9b996
112692025-04-30T06:42:49.093ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112702025-04-30T06:42:49.093ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59776 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 12e5a7a8-d63d-409f-91e2-6cad56c9b996
112712025-04-30T06:42:49.093ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112722025-04-30T06:42:49.093ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
112732025-04-30T06:42:49.094ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112742025-04-30T06:42:49.182ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
112752025-04-30T06:42:49.188ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 88 36 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .6..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51014 req_id = 4012ec49-0b5a-4876-bdba-c580c20eae27
112762025-04-30T06:42:49.188ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34870,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51014 req_id = 4012ec49-0b5a-4876-bdba-c580c20eae27
112772025-04-30T06:42:49.188ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
112782025-04-30T06:42:49.188ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34870, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51014 req_id = 4012ec49-0b5a-4876-bdba-c580c20eae27
112792025-04-30T06:42:49.189ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
112802025-04-30T06:42:49.189ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
112812025-04-30T06:42:49.189ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
112822025-04-30T06:42:49.205ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "85206f8f-ba98-4041-bc9e-c59d0d5099d2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
112832025-04-30T06:42:49.206ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
112842025-04-30T06:42:49.224ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c30b49d9-1aa4-4195-9a18-d1d4e66a9001", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
112852025-04-30T06:42:49.225ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
112862025-04-30T06:42:49.227ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffb66a4f-3f18-4c2a-b777-6abceb87fcfa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
112872025-04-30T06:42:49.227ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
112882025-04-30T06:42:49.234ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7feffccc-cf2b-4c9c-bedf-877ac5444122", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
112892025-04-30T06:42:49.235ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
112902025-04-30T06:42:49.248ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
112912025-04-30T06:42:49.249ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "198386f4-5cb2-485b-b837-847dcd330f12", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
112922025-04-30T06:42:49.250ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
112932025-04-30T06:42:49.268ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25bd719e-ba27-4208-acbf-ce1e83af33e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
112942025-04-30T06:42:49.268ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
112952025-04-30T06:42:49.270ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd28acff-e853-4de7-8dda-15889a171652", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
112962025-04-30T06:42:49.270ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
112972025-04-30T06:42:49.274ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a9379e3-a820-4a46-8abf-7e4b345a380c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
112982025-04-30T06:42:49.274ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
112992025-04-30T06:42:49.278ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3c6242-705b-4b22-9e9b-ac6e13e5ee6e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113002025-04-30T06:42:49.278ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
113012025-04-30T06:42:49.281ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c7ac738-015f-4e29-a16b-23808bd757e2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113022025-04-30T06:42:49.281ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
113032025-04-30T06:42:49.286ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e51e6184-3544-4a1c-b4ec-b68c37e52013", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113042025-04-30T06:42:49.287ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
113052025-04-30T06:42:49.290ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad35dce9-1808-4ad3-b1cb-76e4db68583e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113062025-04-30T06:42:49.291ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
113072025-04-30T06:42:49.293ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97fec8f9-c420-4c1e-ae27-bda777dfeccd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113082025-04-30T06:42:49.293ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
113092025-04-30T06:42:49.299ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113102025-04-30T06:42:49.299ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113112025-04-30T06:42:49.299ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113122025-04-30T06:42:49.299ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
113132025-04-30T06:42:49.300ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113142025-04-30T06:42:49.303ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113152025-04-30T06:42:49.312ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113162025-04-30T06:42:49.313ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113172025-04-30T06:42:49.313ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
113182025-04-30T06:42:49.314ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113192025-04-30T06:42:49.315ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b4fc573-5ffd-4d05-9dd3-92d1634c9b87", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113202025-04-30T06:42:49.316ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
113212025-04-30T06:42:49.319ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e162f4ac-fb3b-49b6-937f-214ecc84cde7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113222025-04-30T06:42:49.319ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
113232025-04-30T06:42:49.321ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113242025-04-30T06:42:49.321ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
113252025-04-30T06:42:49.322ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113262025-04-30T06:42:49.324ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e26fea6d-28a2-4e96-a61d-8cf1a1609121", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113272025-04-30T06:42:49.324ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
113282025-04-30T06:42:49.327ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1ae47d7-147e-4074-a002-69ad200abd0a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113292025-04-30T06:42:49.327ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
113302025-04-30T06:42:49.331ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6d18db5-1e2c-42d2-a543-9571cb24d887", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113312025-04-30T06:42:49.331ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
113322025-04-30T06:42:49.337ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98d1709d-9a41-45ac-914d-93cfb7f501f3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113332025-04-30T06:42:49.338ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
113342025-04-30T06:42:49.350ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b25a2ab1-dee3-49d7-a2cc-8890effe7d41", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113352025-04-30T06:42:49.350ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
113362025-04-30T06:42:49.352ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf9ba92f-070c-4bb6-b40a-edfe2ba1f354", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113372025-04-30T06:42:49.352ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
113382025-04-30T06:42:49.355ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4567b8dc-a782-44fb-b6c1-8580a233a717", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113392025-04-30T06:42:49.355ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
113402025-04-30T06:42:49.359ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f59d753-5ef4-47e8-a93d-fd2dbcbf47fa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113412025-04-30T06:42:49.359ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
113422025-04-30T06:42:49.361ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58ec97d7-e06a-4ca5-b95d-cc9f9210ca65", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113432025-04-30T06:42:49.361ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
113442025-04-30T06:42:49.367ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "223377e5-5abf-4bf1-8e6b-18a45c4fd2c7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113452025-04-30T06:42:49.367ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
113462025-04-30T06:42:49.368ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
113472025-04-30T06:42:49.371ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a386d6c-5136-49b0-a136-12cae8cb33b1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113482025-04-30T06:42:49.371ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
113492025-04-30T06:42:49.373ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8a21373-332b-48f1-bdc6-214134ed4c9a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113502025-04-30T06:42:49.373ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
113512025-04-30T06:42:49.375ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eca271f0-b15e-494e-b484-cf97f5282dcf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113522025-04-30T06:42:49.375ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
113532025-04-30T06:42:49.384ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "447b1cc3-5c9d-4f10-8126-9f5150bf8b76", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113542025-04-30T06:42:49.384ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
113552025-04-30T06:42:49.385ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b8a46e0-5ef4-4442-afba-16d903cac52b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113562025-04-30T06:42:49.386ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
113572025-04-30T06:42:49.389ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bab7c43b-1dc6-4f44-b248-13e3b0086615", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113582025-04-30T06:42:49.389ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
113592025-04-30T06:42:49.394ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84155dc4-a4f2-426f-8cf0-746483ba11cd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113602025-04-30T06:42:49.395ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
113612025-04-30T06:42:49.398ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaa615c1-3f81-43d4-bf89-c2ead6362a9f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113622025-04-30T06:42:49.398ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
113632025-04-30T06:42:49.408ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0b33fe8-b62e-48b7-9a98-3728e160ce43", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113642025-04-30T06:42:49.408ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
113652025-04-30T06:42:49.411ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a37abba-ceac-4e09-a61d-3d30467cdb32", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113662025-04-30T06:42:49.411ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
113672025-04-30T06:42:49.415ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09e36bfd-ef71-4899-91ce-f53d5d6a498f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:48 GMT"} })
113682025-04-30T06:42:49.415ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
113692025-04-30T06:42:49.421ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113702025-04-30T06:42:49.421ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113712025-04-30T06:42:49.421ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113722025-04-30T06:42:49.421ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
113732025-04-30T06:42:49.423ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113742025-04-30T06:42:49.425ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7bdd707-c3c1-4b14-844c-ffd0606e7335", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
113752025-04-30T06:42:49.426ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
113762025-04-30T06:42:49.426ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113772025-04-30T06:42:49.457ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113782025-04-30T06:42:49.457ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113792025-04-30T06:42:49.457ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
113802025-04-30T06:42:49.458ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113812025-04-30T06:42:49.461ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "daa91aae-9590-4658-abb8-eab36e7bdcf0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
113822025-04-30T06:42:49.461ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
113832025-04-30T06:42:49.470ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113842025-04-30T06:42:49.470ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
113852025-04-30T06:42:49.470ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113862025-04-30T06:42:49.479ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05318c44-98ee-4484-9a7a-ca0db70611c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
113872025-04-30T06:42:49.479ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
113882025-04-30T06:42:49.487ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87ac0f8c-ae52-4427-86b2-02f7eb9709b2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
113892025-04-30T06:42:49.487ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
113902025-04-30T06:42:49.490ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
113912025-04-30T06:42:49.493ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113922025-04-30T06:42:49.494ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 269eddc9-1664-4b59-869c-c4c24c14b6fe uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113932025-04-30T06:42:49.494ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 269eddc9-1664-4b59-869c-c4c24c14b6fe resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113942025-04-30T06:42:49.494ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 269eddc9-1664-4b59-869c-c4c24c14b6fe roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113952025-04-30T06:42:49.495ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 269eddc9-1664-4b59-869c-c4c24c14b6fe resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113962025-04-30T06:42:49.498ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "734e2e81-c95f-4b1a-95cc-4fdbeee8d1db", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
113972025-04-30T06:42:49.498ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
113982025-04-30T06:42:49.500ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 5844 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 269eddc9-1664-4b59-869c-c4c24c14b6fe response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113992025-04-30T06:42:49.500ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea3dc4c5-fe6d-46ff-89dd-15ca8be4ba9e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
114002025-04-30T06:42:49.500ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
114012025-04-30T06:42:49.500ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "269eddc9-1664-4b59-869c-c4c24c14b6fe", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
114022025-04-30T06:42:49.500ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
114032025-04-30T06:42:49.500ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1ae495c4-4c1c-42d7-8654-f3bce4939337 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
114042025-04-30T06:42:49.501ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1ae495c4-4c1c-42d7-8654-f3bce4939337 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
114052025-04-30T06:42:49.501ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1ae495c4-4c1c-42d7-8654-f3bce4939337 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
114062025-04-30T06:42:49.501ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1ae495c4-4c1c-42d7-8654-f3bce4939337 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
114072025-04-30T06:42:49.503ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7b50257-98e6-4efb-9b84-a49c273d3323", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
114082025-04-30T06:42:49.504ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
114092025-04-30T06:42:49.504ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 4089 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 1ae495c4-4c1c-42d7-8654-f3bce4939337 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
114102025-04-30T06:42:49.505ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ae495c4-4c1c-42d7-8654-f3bce4939337", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
114112025-04-30T06:42:49.505ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114122025-04-30T06:42:49.505ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
114132025-04-30T06:42:49.505ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
114142025-04-30T06:42:49.510ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa3ee3c4-895e-4366-af29-025f5a11b9be", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
114152025-04-30T06:42:49.510ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
114162025-04-30T06:42:49.510ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
114172025-04-30T06:42:49.514ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec803f1d-4d39-4289-8536-a03391a5329f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
114182025-04-30T06:42:49.515ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114192025-04-30T06:42:49.515ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114202025-04-30T06:42:49.515ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114212025-04-30T06:42:49.515ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114222025-04-30T06:42:49.515ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114232025-04-30T06:42:49.517ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114242025-04-30T06:42:49.549ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
114252025-04-30T06:42:49.549ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114262025-04-30T06:42:49.549ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114272025-04-30T06:42:49.549ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114282025-04-30T06:42:49.549ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114292025-04-30T06:42:49.550ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114302025-04-30T06:42:49.552ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114312025-04-30T06:42:49.552ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114322025-04-30T06:42:49.554ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114332025-04-30T06:42:49.557ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
114342025-04-30T06:42:49.557ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114352025-04-30T06:42:49.557ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114362025-04-30T06:42:49.558ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114372025-04-30T06:42:49.558ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114382025-04-30T06:42:49.561ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114392025-04-30T06:42:49.561ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114402025-04-30T06:42:49.561ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114412025-04-30T06:42:49.561ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114422025-04-30T06:42:49.561ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114432025-04-30T06:42:49.561ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114442025-04-30T06:42:49.561ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
114452025-04-30T06:42:49.562ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114462025-04-30T06:42:49.647ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
114472025-04-30T06:42:49.647ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
114482025-04-30T06:42:49.650ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114492025-04-30T06:42:49.657ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114502025-04-30T06:42:49.657ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114512025-04-30T06:42:49.657ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
114522025-04-30T06:42:49.658ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114532025-04-30T06:42:49.658ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:58221 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
114542025-04-30T06:42:49.659ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
114552025-04-30T06:42:49.659ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
114562025-04-30T06:42:49.660ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:58221 req_id = af3cbe13-e8f2-4385-8ebc-af74206e1b25 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
114572025-04-30T06:42:49.661ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = 127.0.0.1:65039 method = GET remote_addr = 127.0.0.1:55281 req_id = 9730d306-ef69-40e2-9e96-1ca1eb487273 uri = /de3c7c83-8252-43f4-ba2e-cbd92004c4b0
114582025-04-30T06:42:49.674ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 13302 local_addr = [::1]:62607 method = GET remote_addr = [::1]:58221 req_id = af3cbe13-e8f2-4385-8ebc-af74206e1b25 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
114592025-04-30T06:42:49.674ZINFOtest_delete_snapshot (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 13354 local_addr = 127.0.0.1:65039 method = GET remote_addr = 127.0.0.1:55281 req_id = 9730d306-ef69-40e2-9e96-1ca1eb487273 response_code = 200 uri = /de3c7c83-8252-43f4-ba2e-cbd92004c4b0
114602025-04-30T06:42:49.675ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "af3cbe13-e8f2-4385-8ebc-af74206e1b25", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
114612025-04-30T06:42:49.678ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
114622025-04-30T06:42:49.678ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
114632025-04-30T06:42:49.678ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
114642025-04-30T06:42:49.681ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
114652025-04-30T06:42:49.681ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114662025-04-30T06:42:49.681ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114672025-04-30T06:42:49.681ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114682025-04-30T06:42:49.682ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114692025-04-30T06:42:49.682ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
114702025-04-30T06:42:49.690ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
114712025-04-30T06:42:49.690ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
114722025-04-30T06:42:49.690ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
114732025-04-30T06:42:49.690ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114742025-04-30T06:42:49.690ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
114752025-04-30T06:42:49.692ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114762025-04-30T06:42:49.692ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114772025-04-30T06:42:49.697ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114782025-04-30T06:42:49.697ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114792025-04-30T06:42:49.697ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114802025-04-30T06:42:49.699ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114812025-04-30T06:42:49.749ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
114822025-04-30T06:42:49.750ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
114832025-04-30T06:42:49.750ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 694.801888ms iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
114842025-04-30T06:42:49.750ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
114852025-04-30T06:42:49.750ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
114862025-04-30T06:42:49.750ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114872025-04-30T06:42:49.750ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114882025-04-30T06:42:49.750ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114892025-04-30T06:42:49.751ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114902025-04-30T06:42:49.756ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114912025-04-30T06:42:49.760ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114922025-04-30T06:42:49.760ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114932025-04-30T06:42:49.760ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114942025-04-30T06:42:49.760ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114952025-04-30T06:42:49.761ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
114962025-04-30T06:42:49.765ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
114972025-04-30T06:42:49.765ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
114982025-04-30T06:42:49.767ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 43 5d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 C]..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50091 req_id = 0b0e5597-e9cc-4a45-85ee-d515a584b025
114992025-04-30T06:42:49.767ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17245,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50091 req_id = 0b0e5597-e9cc-4a45-85ee-d515a584b025
115002025-04-30T06:42:49.767ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
115012025-04-30T06:42:49.767ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115022025-04-30T06:42:49.767ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50091 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0b0e5597-e9cc-4a45-85ee-d515a584b025
115032025-04-30T06:42:49.768ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4234, tv_nsec: 567128526 } })
115042025-04-30T06:42:49.768ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
115052025-04-30T06:42:49.768ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8b 41 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .A..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62874 req_id = 96205169-4da4-4d28-9822-a1fdecd0baee
115062025-04-30T06:42:49.768ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35649,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62874 req_id = 96205169-4da4-4d28-9822-a1fdecd0baee
115072025-04-30T06:42:49.768ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115082025-04-30T06:42:49.768ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62874 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 96205169-4da4-4d28-9822-a1fdecd0baee
115092025-04-30T06:42:49.833ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
115102025-04-30T06:42:49.834ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115112025-04-30T06:42:49.834ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115122025-04-30T06:42:49.834ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115132025-04-30T06:42:49.834ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
115142025-04-30T06:42:49.835ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115152025-04-30T06:42:49.836ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c4 d1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64175 req_id = 11ba15d8-0198-4935-846d-c7048c75ac7d
115162025-04-30T06:42:49.836ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50385,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64175 req_id = 11ba15d8-0198-4935-846d-c7048c75ac7d
115172025-04-30T06:42:49.837ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
115182025-04-30T06:42:49.837ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50385, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64175 req_id = 11ba15d8-0198-4935-846d-c7048c75ac7d
115192025-04-30T06:42:49.838ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
115202025-04-30T06:42:49.838ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
115212025-04-30T06:42:49.838ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
115222025-04-30T06:42:49.839ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115232025-04-30T06:42:49.851ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115242025-04-30T06:42:49.852ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115252025-04-30T06:42:49.852ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
115262025-04-30T06:42:49.853ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115272025-04-30T06:42:49.855ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d52b0b0-723e-445d-829a-75e7c3bd27c9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115282025-04-30T06:42:49.856ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
115292025-04-30T06:42:49.859ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05b911e5-516d-440f-af52-f9a24c385a7f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115302025-04-30T06:42:49.860ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
115312025-04-30T06:42:49.860ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115322025-04-30T06:42:49.860ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
115332025-04-30T06:42:49.862ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115342025-04-30T06:42:49.866ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c0e57f0-f014-4fbf-8d23-5a7cfcbfd92b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115352025-04-30T06:42:49.866ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
115362025-04-30T06:42:49.873ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4089f8ae-4c94-4918-adeb-ea57f612732e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115372025-04-30T06:42:49.873ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
115382025-04-30T06:42:49.877ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4afb14f6-eee1-4ff1-bc3f-399835aa103d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115392025-04-30T06:42:49.878ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
115402025-04-30T06:42:49.908ZDEBGtest_delete_snapshot (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
115412025-04-30T06:42:49.908ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
115422025-04-30T06:42:49.908ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcb7397b-b91d-44e8-9c31-6645f7162316", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115432025-04-30T06:42:49.908ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
115442025-04-30T06:42:49.908ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
115452025-04-30T06:42:49.912ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
115462025-04-30T06:42:49.920ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 12 table_name = fields_string
115472025-04-30T06:42:49.921ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
115482025-04-30T06:42:49.921ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
115492025-04-30T06:42:49.924ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee578c7a-acb1-44fd-980f-0c6c7d63a5d5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115502025-04-30T06:42:49.924ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
115512025-04-30T06:42:49.925ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 5 table_name = fields_u16
115522025-04-30T06:42:49.925ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
115532025-04-30T06:42:49.928ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 5 table_name = fields_uuid
115542025-04-30T06:42:49.928ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
115552025-04-30T06:42:49.930ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7c1501a-c1b6-4688-8572-f2903b69c71f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115562025-04-30T06:42:49.931ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
115572025-04-30T06:42:49.932ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1da547c0-26a2-41de-8250-9799b1f3f89a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115582025-04-30T06:42:49.932ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
115592025-04-30T06:42:49.934ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 5 table_name = measurements_histogramu64
115602025-04-30T06:42:49.934ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
115612025-04-30T06:42:49.935ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "967c14a7-0802-4607-ae8c-fbd6e44079c3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115622025-04-30T06:42:49.935ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
115632025-04-30T06:42:49.936ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
115642025-04-30T06:42:49.936ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
115652025-04-30T06:42:49.938ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff3dc344-8b7f-4123-9f95-1cf87c5a5a34", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115662025-04-30T06:42:49.938ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
115672025-04-30T06:42:49.938ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a0c6570-2022-49be-8818-74d19b7741c5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115682025-04-30T06:42:49.938ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
115692025-04-30T06:42:49.940ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d9b8516-4736-4e3a-b9ff-8e8e7ca1bede", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115702025-04-30T06:42:49.940ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
115712025-04-30T06:42:49.942ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2a54512-2eef-42e1-82c5-53e4cf2e9dc5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115722025-04-30T06:42:49.943ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
115732025-04-30T06:42:49.945ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4cca907-20fd-407e-b2b0-a330453ea175", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115742025-04-30T06:42:49.945ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
115752025-04-30T06:42:49.947ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fddbfbee-b24f-4291-85ef-0f6a3a9fd3fb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115762025-04-30T06:42:49.947ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
115772025-04-30T06:42:49.950ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a198eae-6b8f-4493-b5cd-5d51f4a6cded", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115782025-04-30T06:42:49.950ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
115792025-04-30T06:42:49.952ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21202163-acb2-469e-83f9-5be196dc9c88", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115802025-04-30T06:42:49.952ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
115812025-04-30T06:42:49.954ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14b5550b-aad3-45c7-995b-8864932b44f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115822025-04-30T06:42:49.954ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
115832025-04-30T06:42:49.958ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09e62527-557d-4a44-ba17-10d727bd7ba1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115842025-04-30T06:42:49.958ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
115852025-04-30T06:42:49.958ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
115862025-04-30T06:42:49.959ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 57 99 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 W............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59712 req_id = 7ecc5bcd-8a94-4438-a80b-7761a9dcfcc1
115872025-04-30T06:42:49.959ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22425,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59712 req_id = 7ecc5bcd-8a94-4438-a80b-7761a9dcfcc1
115882025-04-30T06:42:49.959ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
115892025-04-30T06:42:49.959ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115902025-04-30T06:42:49.959ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59712 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7ecc5bcd-8a94-4438-a80b-7761a9dcfcc1
115912025-04-30T06:42:49.960ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4234, tv_nsec: 759059306 } })
115922025-04-30T06:42:49.960ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
115932025-04-30T06:42:49.960ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71364673-6d37-4639-9fbd-31beb4f2ffe0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
115942025-04-30T06:42:49.960ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
115952025-04-30T06:42:49.963ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 47 c6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e G...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58103 req_id = abf29592-201d-4999-9ef1-4966b55f0b96
115962025-04-30T06:42:49.963ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18374,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58103 req_id = abf29592-201d-4999-9ef1-4966b55f0b96
115972025-04-30T06:42:49.963ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115982025-04-30T06:42:49.963ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58103 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = abf29592-201d-4999-9ef1-4966b55f0b96
115992025-04-30T06:42:49.964ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
116002025-04-30T06:42:50.062ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
116012025-04-30T06:42:50.063ZDEBGtest_delete_snapshot (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
116022025-04-30T06:42:50.063ZDEBGtest_delete_snapshot (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
116032025-04-30T06:42:50.063ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
116042025-04-30T06:42:50.065ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
116052025-04-30T06:42:50.065ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116062025-04-30T06:42:50.065ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116072025-04-30T06:42:50.065ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116082025-04-30T06:42:50.065ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
116092025-04-30T06:42:50.066ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116102025-04-30T06:42:50.066ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
116112025-04-30T06:42:50.067ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2a44838-c2b2-47ea-8c98-0c372bc25c35", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116122025-04-30T06:42:50.067ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
116132025-04-30T06:42:50.068ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
116142025-04-30T06:42:50.071ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 8f 76 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .v..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55324 req_id = 55d282af-d5b5-4d21-9095-d40ba0008160
116152025-04-30T06:42:50.072ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36726,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55324 req_id = 55d282af-d5b5-4d21-9095-d40ba0008160
116162025-04-30T06:42:50.072ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
116172025-04-30T06:42:50.072ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116182025-04-30T06:42:50.072ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55324 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 55d282af-d5b5-4d21-9095-d40ba0008160
116192025-04-30T06:42:50.073ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116202025-04-30T06:42:50.080ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116212025-04-30T06:42:50.080ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116222025-04-30T06:42:50.081ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
116232025-04-30T06:42:50.081ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116242025-04-30T06:42:50.082ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4234, tv_nsec: 881097539 } })
116252025-04-30T06:42:50.082ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
116262025-04-30T06:42:50.090ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 19 0c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63530 req_id = b77c7e09-72fe-4d2a-b66b-cc1ec10ceec9
116272025-04-30T06:42:50.090ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6412,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63530 req_id = b77c7e09-72fe-4d2a-b66b-cc1ec10ceec9
116282025-04-30T06:42:50.090ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116292025-04-30T06:42:50.090ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63530 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b77c7e09-72fe-4d2a-b66b-cc1ec10ceec9
116302025-04-30T06:42:50.091ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6587ba38-b7ed-4e8e-80f7-c85c2603cc32", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116312025-04-30T06:42:50.091ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
116322025-04-30T06:42:50.091ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116332025-04-30T06:42:50.091ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
116342025-04-30T06:42:50.092ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116352025-04-30T06:42:50.092ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
116362025-04-30T06:42:50.193ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
116372025-04-30T06:42:50.193ZDEBGtest_delete_snapshot (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
116382025-04-30T06:42:50.193ZDEBGtest_delete_snapshot (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
116392025-04-30T06:42:50.193ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
116402025-04-30T06:42:50.196ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
116412025-04-30T06:42:50.235ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
116422025-04-30T06:42:50.243ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: f8 80 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61820 req_id = 55baecc8-cc64-4d3a-8af0-f1272f062118
116432025-04-30T06:42:50.243ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63616,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61820 req_id = 55baecc8-cc64-4d3a-8af0-f1272f062118
116442025-04-30T06:42:50.243ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
116452025-04-30T06:42:50.243ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116462025-04-30T06:42:50.243ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61820 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 55baecc8-cc64-4d3a-8af0-f1272f062118
116472025-04-30T06:42:50.244ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4235, tv_nsec: 43592370 } })
116482025-04-30T06:42:50.244ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
116492025-04-30T06:42:50.249ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a8 f2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50275 req_id = 757b0690-1862-4f3c-b59c-41691eb3c541
116502025-04-30T06:42:50.249ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43250,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50275 req_id = 757b0690-1862-4f3c-b59c-41691eb3c541
116512025-04-30T06:42:50.249ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116522025-04-30T06:42:50.249ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50275 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 757b0690-1862-4f3c-b59c-41691eb3c541
116532025-04-30T06:42:50.250ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
116542025-04-30T06:42:50.325ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
116552025-04-30T06:42:50.327ZDEBGtest_delete_snapshot (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
116562025-04-30T06:42:50.327ZDEBGtest_delete_snapshot (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
116572025-04-30T06:42:50.327ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
116582025-04-30T06:42:50.329ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
116592025-04-30T06:42:50.330ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "248f848f-a528-4c80-8305-97c5dc7837a4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116602025-04-30T06:42:50.330ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
116612025-04-30T06:42:50.347ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7cbebc8-a6e9-4456-a731-e479bdcf67af", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116622025-04-30T06:42:50.347ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
116632025-04-30T06:42:50.351ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "538011c9-6663-43cd-af63-64a9901e19df", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116642025-04-30T06:42:50.351ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
116652025-04-30T06:42:50.357ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
116662025-04-30T06:42:50.360ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9e9cc0f-de37-4bde-8332-611288a6be95", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116672025-04-30T06:42:50.360ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
116682025-04-30T06:42:50.362ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26549f1d-1bac-4255-b3c5-6c53d101a13a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116692025-04-30T06:42:50.362ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
116702025-04-30T06:42:50.366ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a4419ef-aaeb-48db-b5ab-9af2e37395db", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116712025-04-30T06:42:50.366ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
116722025-04-30T06:42:50.384ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "791d6b45-1303-40b9-a857-d470c4070e7c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116732025-04-30T06:42:50.384ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
116742025-04-30T06:42:50.385ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72770c32-e633-41c9-8209-bec5bcb23dbf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116752025-04-30T06:42:50.385ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
116762025-04-30T06:42:50.386ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1d07c0a-d068-49f6-af8f-cc0041cde4b0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116772025-04-30T06:42:50.387ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
116782025-04-30T06:42:50.387ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c34c13be-53f0-4b86-9bba-b0fcd25df1ac", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116792025-04-30T06:42:50.388ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
116802025-04-30T06:42:50.388ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db9c2ac9-bcfc-4022-9d82-390d0acab3f4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116812025-04-30T06:42:50.388ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
116822025-04-30T06:42:50.396ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "797e9e3f-dac4-44ff-b2b0-3e8ef5082623", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116832025-04-30T06:42:50.396ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
116842025-04-30T06:42:50.398ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90f09bb6-6faf-48ea-a598-d5443371f0e3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116852025-04-30T06:42:50.398ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
116862025-04-30T06:42:50.400ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f12f393-2b87-403a-a5c3-2a5aaf168dc0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116872025-04-30T06:42:50.400ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
116882025-04-30T06:42:50.404ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56765446-f63a-475c-b8c8-dddb601b9cfe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116892025-04-30T06:42:50.404ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
116902025-04-30T06:42:50.421ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116912025-04-30T06:42:50.421ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116922025-04-30T06:42:50.421ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116932025-04-30T06:42:50.421ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
116942025-04-30T06:42:50.422ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116952025-04-30T06:42:50.425ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9acc7fd-cb0c-4e7b-97d4-1edee3432ea7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:49 GMT"} })
116962025-04-30T06:42:50.425ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
116972025-04-30T06:42:50.425ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116982025-04-30T06:42:50.430ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116992025-04-30T06:42:50.430ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117002025-04-30T06:42:50.430ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
117012025-04-30T06:42:50.432ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117022025-04-30T06:42:50.436ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117032025-04-30T06:42:50.436ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
117042025-04-30T06:42:50.436ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117052025-04-30T06:42:50.440ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89077a00-e339-498a-9f1e-6116650b3cba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
117062025-04-30T06:42:50.440ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
117072025-04-30T06:42:50.446ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e26fba7-37e5-430c-b5f6-6e06efc114f0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
117082025-04-30T06:42:50.447ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
117092025-04-30T06:42:50.449ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c33b3a24-4ec6-4229-a52d-db50b452eb8b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
117102025-04-30T06:42:50.449ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
117112025-04-30T06:42:50.450ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2291191a-b18b-4716-b4b0-817f541b7a39", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
117122025-04-30T06:42:50.450ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
117132025-04-30T06:42:50.452ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddb0ee96-f69e-4fc8-9907-98b843a57e37", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
117142025-04-30T06:42:50.452ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
117152025-04-30T06:42:50.453ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba6cd81b-78fe-4f03-bfc1-a541a75535c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
117162025-04-30T06:42:50.454ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117172025-04-30T06:42:50.454ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117182025-04-30T06:42:50.454ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117192025-04-30T06:42:50.454ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117202025-04-30T06:42:50.454ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117212025-04-30T06:42:50.470ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117222025-04-30T06:42:50.479ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117232025-04-30T06:42:50.479ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117242025-04-30T06:42:50.479ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117252025-04-30T06:42:50.479ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117262025-04-30T06:42:50.479ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117272025-04-30T06:42:50.490ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117282025-04-30T06:42:50.515ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117292025-04-30T06:42:50.521ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117302025-04-30T06:42:50.521ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117312025-04-30T06:42:50.521ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117322025-04-30T06:42:50.524ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117332025-04-30T06:42:50.634ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
117342025-04-30T06:42:50.635ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
117352025-04-30T06:42:50.636ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:59670 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
117362025-04-30T06:42:50.636ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117372025-04-30T06:42:50.636ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117382025-04-30T06:42:50.638ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:59670 req_id = 86386940-032d-4011-b2a3-45d12229b2e2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117392025-04-30T06:42:50.640ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 2743 local_addr = [::1]:62607 method = GET remote_addr = [::1]:59670 req_id = 86386940-032d-4011-b2a3-45d12229b2e2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117402025-04-30T06:42:50.644ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "86386940-032d-4011-b2a3-45d12229b2e2", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
117412025-04-30T06:42:50.649ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117422025-04-30T06:42:50.649ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117432025-04-30T06:42:50.649ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
117442025-04-30T06:42:50.653ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117452025-04-30T06:42:50.653ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117462025-04-30T06:42:50.653ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117472025-04-30T06:42:50.653ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117482025-04-30T06:42:50.654ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117492025-04-30T06:42:50.654ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117502025-04-30T06:42:50.664ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117512025-04-30T06:42:50.664ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117522025-04-30T06:42:50.664ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117532025-04-30T06:42:50.666ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
117542025-04-30T06:42:50.671ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117552025-04-30T06:42:50.684ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117562025-04-30T06:42:50.684ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117572025-04-30T06:42:50.684ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117582025-04-30T06:42:50.685ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117592025-04-30T06:42:50.719ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117602025-04-30T06:42:50.719ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117612025-04-30T06:42:50.719ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117622025-04-30T06:42:50.720ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
117632025-04-30T06:42:50.721ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117642025-04-30T06:42:50.736ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117652025-04-30T06:42:50.746ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117662025-04-30T06:42:50.746ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117672025-04-30T06:42:50.746ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
117682025-04-30T06:42:50.748ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117692025-04-30T06:42:50.845ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
117702025-04-30T06:42:50.845ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
117712025-04-30T06:42:50.845ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.095333911s iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
117722025-04-30T06:42:50.845ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
117732025-04-30T06:42:50.845ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117742025-04-30T06:42:50.845ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117752025-04-30T06:42:50.845ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117762025-04-30T06:42:50.845ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117772025-04-30T06:42:50.846ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117782025-04-30T06:42:50.856ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117792025-04-30T06:42:50.856ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
117802025-04-30T06:42:50.857ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117812025-04-30T06:42:50.857ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117822025-04-30T06:42:50.862ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117832025-04-30T06:42:50.862ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117842025-04-30T06:42:50.862ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117852025-04-30T06:42:50.862ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117862025-04-30T06:42:50.878ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
117872025-04-30T06:42:50.879ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
117882025-04-30T06:42:50.881ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f8 5e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .^..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55635 req_id = f55ffc20-03c5-44fc-84f5-af404f054b2a
117892025-04-30T06:42:50.882ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63582,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55635 req_id = f55ffc20-03c5-44fc-84f5-af404f054b2a
117902025-04-30T06:42:50.882ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
117912025-04-30T06:42:50.884ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117922025-04-30T06:42:50.886ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55635 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f55ffc20-03c5-44fc-84f5-af404f054b2a
117932025-04-30T06:42:50.888ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4235, tv_nsec: 687772847 } })
117942025-04-30T06:42:50.889ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
117952025-04-30T06:42:50.896ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ba 18 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59208 req_id = ee39be60-2a7e-47cb-a2f4-d5d16ad63e7e
117962025-04-30T06:42:50.896ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47640,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59208 req_id = ee39be60-2a7e-47cb-a2f4-d5d16ad63e7e
117972025-04-30T06:42:50.896ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117982025-04-30T06:42:50.896ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59208 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ee39be60-2a7e-47cb-a2f4-d5d16ad63e7e
117992025-04-30T06:42:50.962ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
118002025-04-30T06:42:50.962ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
118012025-04-30T06:42:50.962ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
118022025-04-30T06:42:50.962ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
118032025-04-30T06:42:50.964ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
118042025-04-30T06:42:50.964ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ba 40 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .@..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57270 req_id = b027cb38-857f-49e3-83ac-901bead4014b
118052025-04-30T06:42:50.965ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47680,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57270 req_id = b027cb38-857f-49e3-83ac-901bead4014b
118062025-04-30T06:42:50.965ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
118072025-04-30T06:42:50.965ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47680, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57270 req_id = b027cb38-857f-49e3-83ac-901bead4014b
118082025-04-30T06:42:50.965ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
118092025-04-30T06:42:50.965ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
118102025-04-30T06:42:50.965ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
118112025-04-30T06:42:50.970ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
118122025-04-30T06:42:50.970ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
118132025-04-30T06:42:50.972ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "af7434cf-1239-4b66-98b4-0899aa6a34e7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118142025-04-30T06:42:50.972ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
118152025-04-30T06:42:50.974ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da6ae84e-d88a-40ff-806b-f58d3f97d091", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118162025-04-30T06:42:50.975ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
118172025-04-30T06:42:50.979ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
118182025-04-30T06:42:50.979ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
118192025-04-30T06:42:50.999ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38148cca-9a27-4dcb-b586-36fb938de28e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118202025-04-30T06:42:50.999ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
118212025-04-30T06:42:51.017ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
118222025-04-30T06:42:51.022ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d95e376-9226-4d30-abee-08fb70829e95", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118232025-04-30T06:42:51.023ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
118242025-04-30T06:42:51.027ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b676796b-4491-46dd-81ee-fbc6d4ea8544", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118252025-04-30T06:42:51.027ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
118262025-04-30T06:42:51.030ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbe868be-7d1e-4efd-bf1d-63f4c167ea88", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118272025-04-30T06:42:51.031ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
118282025-04-30T06:42:51.048ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd293d59-2b2e-496a-81c8-ad211f07bf77", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118292025-04-30T06:42:51.048ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
118302025-04-30T06:42:51.053ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfa3118e-5569-48c2-a403-d92f9f32675b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118312025-04-30T06:42:51.053ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
118322025-04-30T06:42:51.054ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "010e1c7f-3252-4be9-8165-f1b5c83e4eef", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118332025-04-30T06:42:51.054ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
118342025-04-30T06:42:51.055ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "579db171-8be5-417f-9c52-039724664b19", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118352025-04-30T06:42:51.056ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
118362025-04-30T06:42:51.058ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0155aa1d-20dc-4155-bd5a-2ae682c290a5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118372025-04-30T06:42:51.058ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
118382025-04-30T06:42:51.064ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4da676ff-c4f9-4aff-b002-be5a83e96988", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118392025-04-30T06:42:51.064ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
118402025-04-30T06:42:51.065ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
118412025-04-30T06:42:51.069ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118422025-04-30T06:42:51.069ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118432025-04-30T06:42:51.069ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118442025-04-30T06:42:51.069ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
118452025-04-30T06:42:51.070ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118462025-04-30T06:42:51.093ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118472025-04-30T06:42:51.097ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118482025-04-30T06:42:51.097ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118492025-04-30T06:42:51.097ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
118502025-04-30T06:42:51.098ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118512025-04-30T06:42:51.098ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b449adb-f50e-49f9-8f46-f5ef6bf976e7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118522025-04-30T06:42:51.098ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
118532025-04-30T06:42:51.109ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118542025-04-30T06:42:51.109ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
118552025-04-30T06:42:51.110ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118562025-04-30T06:42:51.124ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a1bbe30-4bb0-42a8-b6c0-c255ade4c73b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118572025-04-30T06:42:51.124ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
118582025-04-30T06:42:51.131ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fff65988-664c-43cb-a1ab-dbb757f9f1cf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118592025-04-30T06:42:51.141ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
118602025-04-30T06:42:51.144ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fce999d1-9747-4370-9855-277d6d19950d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118612025-04-30T06:42:51.145ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
118622025-04-30T06:42:51.148ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5b7e6df-009d-4180-b075-7d90f7e3e760", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118632025-04-30T06:42:51.148ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
118642025-04-30T06:42:51.155ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
118652025-04-30T06:42:51.155ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed573a19-3eca-42c9-8023-48013e127718", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118662025-04-30T06:42:51.155ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
118672025-04-30T06:42:51.157ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62b4376a-bbeb-4cd2-b0bc-f25a8bc96f80", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118682025-04-30T06:42:51.157ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
118692025-04-30T06:42:51.161ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e21cd88a-470b-48bf-a5a1-8e58a23aa121", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118702025-04-30T06:42:51.161ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
118712025-04-30T06:42:51.162ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54e4e239-b95b-48c7-8cb8-35841726f535", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118722025-04-30T06:42:51.162ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
118732025-04-30T06:42:51.163ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5469c89f-d767-4c9f-ad1b-23c310bc7ba6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118742025-04-30T06:42:51.164ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
118752025-04-30T06:42:51.165ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59343cde-c616-4494-b3ec-c81a32adeefd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118762025-04-30T06:42:51.165ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
118772025-04-30T06:42:51.166ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b9733f-3a00-4493-929e-2575256a2182", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118782025-04-30T06:42:51.166ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
118792025-04-30T06:42:51.168ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ee3f56a-2fc9-4c52-a2e2-fb2a0257081c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118802025-04-30T06:42:51.168ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
118812025-04-30T06:42:51.170ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a9abea0-abd8-41f7-9213-b6ad0f34d3b6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118822025-04-30T06:42:51.171ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
118832025-04-30T06:42:51.187ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7611dc02-421e-479b-bee8-623937d10f4f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118842025-04-30T06:42:51.189ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
118852025-04-30T06:42:51.195ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85f853df-5635-46bd-bf24-29538c7209bc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118862025-04-30T06:42:51.195ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
118872025-04-30T06:42:51.199ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21a8fee7-fb9e-410b-ab1f-220ab682d2bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118882025-04-30T06:42:51.200ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
118892025-04-30T06:42:51.203ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4dd17df-eb19-4552-be08-8b492a4f3027", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118902025-04-30T06:42:51.203ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
118912025-04-30T06:42:51.206ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118922025-04-30T06:42:51.206ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118932025-04-30T06:42:51.206ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118942025-04-30T06:42:51.206ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
118952025-04-30T06:42:51.207ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118962025-04-30T06:42:51.207ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1493a801-e5c8-42cc-802a-616f548d7e70", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118972025-04-30T06:42:51.207ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
118982025-04-30T06:42:51.209ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88211758-c985-4a7e-94d7-60b5a14ad548", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
118992025-04-30T06:42:51.209ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
119002025-04-30T06:42:51.210ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119012025-04-30T06:42:51.223ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119022025-04-30T06:42:51.223ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119032025-04-30T06:42:51.223ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
119042025-04-30T06:42:51.224ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119052025-04-30T06:42:51.229ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06d172a7-4dda-42c6-8530-6b6e2321860a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119062025-04-30T06:42:51.229ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
119072025-04-30T06:42:51.232ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119082025-04-30T06:42:51.232ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
119092025-04-30T06:42:51.234ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119102025-04-30T06:42:51.237ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7e8755c-20d2-4ff6-8fcc-ca2eac0a0ee4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119112025-04-30T06:42:51.237ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
119122025-04-30T06:42:51.238ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21384af2-6e22-4991-8a6e-8b0f5b25ef9f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119132025-04-30T06:42:51.239ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
119142025-04-30T06:42:51.241ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cded92a-55ff-45b6-b9be-8aed2e8382d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119152025-04-30T06:42:51.241ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
119162025-04-30T06:42:51.243ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de97ab81-6974-4ad4-bfe5-eb89b0ad54f7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119172025-04-30T06:42:51.243ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
119182025-04-30T06:42:51.247ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0b425da-bfbb-48c4-8222-e2660dc9b774", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119192025-04-30T06:42:51.248ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
119202025-04-30T06:42:51.256ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8b7f2dc-f094-4e15-857b-3c93ad55a4c2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119212025-04-30T06:42:51.256ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
119222025-04-30T06:42:51.260ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed413fa8-21b5-4ef5-8a2b-385d03d7ab76", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119232025-04-30T06:42:51.275ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
119242025-04-30T06:42:51.283ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38acc33e-3a0c-4518-8f84-ecdea9bff030", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119252025-04-30T06:42:51.283ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
119262025-04-30T06:42:51.283ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1066675e-d7ed-4c1e-a532-d952b9ae1f75", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119272025-04-30T06:42:51.283ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
119282025-04-30T06:42:51.285ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1a2d3e4-67d6-43ba-99f9-09132a780de8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119292025-04-30T06:42:51.286ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
119302025-04-30T06:42:51.297ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
119312025-04-30T06:42:51.304ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15c1b172-920f-41da-a3bb-fc1eabc7f197", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119322025-04-30T06:42:51.305ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
119332025-04-30T06:42:51.310ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "040dfc31-ce21-438d-a1df-60752edb1765", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:50 GMT"} })
119342025-04-30T06:42:51.310ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119352025-04-30T06:42:51.310ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119362025-04-30T06:42:51.310ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119372025-04-30T06:42:51.310ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119382025-04-30T06:42:51.311ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119392025-04-30T06:42:51.315ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119402025-04-30T06:42:51.319ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119412025-04-30T06:42:51.319ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119422025-04-30T06:42:51.319ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119432025-04-30T06:42:51.319ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119442025-04-30T06:42:51.319ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119452025-04-30T06:42:51.320ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119462025-04-30T06:42:51.327ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119472025-04-30T06:42:51.334ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119482025-04-30T06:42:51.334ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119492025-04-30T06:42:51.334ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119502025-04-30T06:42:51.335ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119512025-04-30T06:42:51.451ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
119522025-04-30T06:42:51.451ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
119532025-04-30T06:42:51.452ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119542025-04-30T06:42:51.452ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119552025-04-30T06:42:51.452ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119562025-04-30T06:42:51.452ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
119572025-04-30T06:42:51.455ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119582025-04-30T06:42:51.455ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:64862 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
119592025-04-30T06:42:51.455ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:64862 req_id = 63ef8e66-912d-4b25-bb9e-ad9842b74629 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119602025-04-30T06:42:51.455ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 451 local_addr = [::1]:62607 method = GET remote_addr = [::1]:64862 req_id = 63ef8e66-912d-4b25-bb9e-ad9842b74629 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119612025-04-30T06:42:51.456ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "63ef8e66-912d-4b25-bb9e-ad9842b74629", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
119622025-04-30T06:42:51.456ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119632025-04-30T06:42:51.456ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119642025-04-30T06:42:51.457ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
119652025-04-30T06:42:51.459ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119662025-04-30T06:42:51.459ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119672025-04-30T06:42:51.459ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119682025-04-30T06:42:51.459ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119692025-04-30T06:42:51.460ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119702025-04-30T06:42:51.460ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119712025-04-30T06:42:51.481ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119722025-04-30T06:42:51.481ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119732025-04-30T06:42:51.481ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
119742025-04-30T06:42:51.481ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119752025-04-30T06:42:51.484ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119762025-04-30T06:42:51.484ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
119772025-04-30T06:42:51.485ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119782025-04-30T06:42:51.486ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119792025-04-30T06:42:51.491ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
119802025-04-30T06:42:51.491ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119812025-04-30T06:42:51.491ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119822025-04-30T06:42:51.491ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119832025-04-30T06:42:51.493ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
119842025-04-30T06:42:51.494ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119852025-04-30T06:42:51.495ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 6eb33fda-013a-4f26-b36d-686d2c37d094 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119862025-04-30T06:42:51.496ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 6eb33fda-013a-4f26-b36d-686d2c37d094 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119872025-04-30T06:42:51.496ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 6eb33fda-013a-4f26-b36d-686d2c37d094 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119882025-04-30T06:42:51.497ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 6eb33fda-013a-4f26-b36d-686d2c37d094 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119892025-04-30T06:42:51.502ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 7371 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 6eb33fda-013a-4f26-b36d-686d2c37d094 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119902025-04-30T06:42:51.504ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eb33fda-013a-4f26-b36d-686d2c37d094", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
119912025-04-30T06:42:51.505ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
119922025-04-30T06:42:51.506ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2852a90b-84d4-4d91-ae77-f53861085625 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
119932025-04-30T06:42:51.506ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2852a90b-84d4-4d91-ae77-f53861085625 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
119942025-04-30T06:42:51.506ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2852a90b-84d4-4d91-ae77-f53861085625 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
119952025-04-30T06:42:51.507ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2852a90b-84d4-4d91-ae77-f53861085625 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
119962025-04-30T06:42:51.520ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 13676 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2852a90b-84d4-4d91-ae77-f53861085625 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
119972025-04-30T06:42:51.522ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2852a90b-84d4-4d91-ae77-f53861085625", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
119982025-04-30T06:42:51.522ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119992025-04-30T06:42:51.522ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
120002025-04-30T06:42:51.522ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
120012025-04-30T06:42:51.537ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
120022025-04-30T06:42:51.537ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
120032025-04-30T06:42:51.538ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
120042025-04-30T06:42:51.538ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 692.440242ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
120052025-04-30T06:42:51.538ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
120062025-04-30T06:42:51.538ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120072025-04-30T06:42:51.538ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120082025-04-30T06:42:51.538ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120092025-04-30T06:42:51.538ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120102025-04-30T06:42:51.539ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120112025-04-30T06:42:51.549ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120122025-04-30T06:42:51.552ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120132025-04-30T06:42:51.552ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120142025-04-30T06:42:51.553ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120152025-04-30T06:42:51.553ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120162025-04-30T06:42:51.553ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120172025-04-30T06:42:51.553ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120182025-04-30T06:42:51.555ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120192025-04-30T06:42:51.556ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120202025-04-30T06:42:51.556ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120212025-04-30T06:42:51.556ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120222025-04-30T06:42:51.556ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
120232025-04-30T06:42:51.556ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
120242025-04-30T06:42:51.557ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 87 c6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63753 req_id = a1bd9223-dde8-433f-89e0-53750a6fd8bb
120252025-04-30T06:42:51.557ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34758,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63753 req_id = a1bd9223-dde8-433f-89e0-53750a6fd8bb
120262025-04-30T06:42:51.557ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
120272025-04-30T06:42:51.557ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120282025-04-30T06:42:51.557ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63753 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a1bd9223-dde8-433f-89e0-53750a6fd8bb
120292025-04-30T06:42:51.558ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4236, tv_nsec: 357168134 } })
120302025-04-30T06:42:51.558ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
120312025-04-30T06:42:51.558ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 17 16 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56493 req_id = ab362ff4-f5ca-4124-a8bf-7fab02dd2a68
120322025-04-30T06:42:51.558ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5910,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56493 req_id = ab362ff4-f5ca-4124-a8bf-7fab02dd2a68
120332025-04-30T06:42:51.558ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120342025-04-30T06:42:51.558ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56493 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ab362ff4-f5ca-4124-a8bf-7fab02dd2a68
120352025-04-30T06:42:51.602ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
120362025-04-30T06:42:51.602ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120372025-04-30T06:42:51.602ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120382025-04-30T06:42:51.602ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120392025-04-30T06:42:51.602ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
120402025-04-30T06:42:51.602ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120412025-04-30T06:42:51.603ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: cc 6f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .o..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55495 req_id = 25a77526-b949-4de4-90fa-b497d35d44b5
120422025-04-30T06:42:51.603ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52335,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55495 req_id = 25a77526-b949-4de4-90fa-b497d35d44b5
120432025-04-30T06:42:51.603ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
120442025-04-30T06:42:51.603ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52335, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55495 req_id = 25a77526-b949-4de4-90fa-b497d35d44b5
120452025-04-30T06:42:51.603ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
120462025-04-30T06:42:51.603ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
120472025-04-30T06:42:51.603ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
120482025-04-30T06:42:51.605ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120492025-04-30T06:42:51.610ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120502025-04-30T06:42:51.610ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120512025-04-30T06:42:51.610ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
120522025-04-30T06:42:51.611ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120532025-04-30T06:42:51.614ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e438b8db-0f3c-41a1-97db-826a22e0ad3b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120542025-04-30T06:42:51.614ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
120552025-04-30T06:42:51.615ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120562025-04-30T06:42:51.615ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
120572025-04-30T06:42:51.615ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120582025-04-30T06:42:51.617ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46a20cfb-9590-48b3-9ae3-4fa2bc1506a9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120592025-04-30T06:42:51.618ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
120602025-04-30T06:42:51.630ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdc1c37c-c609-4b60-b1a4-b2b96c8c5fb9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120612025-04-30T06:42:51.630ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
120622025-04-30T06:42:51.634ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06348efa-af63-42fb-9046-a36b40036323", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120632025-04-30T06:42:51.634ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
120642025-04-30T06:42:51.637ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a7e8147-869d-4b24-bdc2-dbb7a775914f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120652025-04-30T06:42:51.637ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
120662025-04-30T06:42:51.647ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9db43df2-9060-47b6-b622-0549797adaaf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120672025-04-30T06:42:51.648ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
120682025-04-30T06:42:51.652ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "480b2356-ada3-4962-80f7-436b4b311f46", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120692025-04-30T06:42:51.652ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
120702025-04-30T06:42:51.662ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
120712025-04-30T06:42:51.664ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "caac25e0-1572-48fe-90b7-8633b8a7e953", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120722025-04-30T06:42:51.665ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
120732025-04-30T06:42:51.666ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dda9db73-3762-4001-9bf9-109a62e3974d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120742025-04-30T06:42:51.666ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
120752025-04-30T06:42:51.667ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a6e188b-a35a-4bdf-a425-976b30f605fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120762025-04-30T06:42:51.667ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
120772025-04-30T06:42:51.668ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f612048-0519-43d1-a40d-d7eba1e99342", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120782025-04-30T06:42:51.668ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
120792025-04-30T06:42:51.670ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e73443b7-8e01-4f65-bc76-b79fe15d9217", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120802025-04-30T06:42:51.670ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
120812025-04-30T06:42:51.674ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "796dd9f3-47c7-40a7-ac83-304f139762d0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120822025-04-30T06:42:51.675ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
120832025-04-30T06:42:51.681ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f90d1aa-f9ec-42a7-b0f8-1cc166fc1f66", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120842025-04-30T06:42:51.681ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
120852025-04-30T06:42:51.683ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c7983cd-63b3-47db-b226-c1a016ab2904", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120862025-04-30T06:42:51.683ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
120872025-04-30T06:42:51.686ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e57bc2bd-5030-4952-8e97-a480687b7dd5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120882025-04-30T06:42:51.686ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
120892025-04-30T06:42:51.690ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b229b971-692e-4b7a-a761-fef028c3f389", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120902025-04-30T06:42:51.699ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
120912025-04-30T06:42:51.703ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad6a130e-d62c-4ec9-b09b-8a596d81d427", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120922025-04-30T06:42:51.703ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
120932025-04-30T06:42:51.707ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8905c6fd-edac-477c-93f9-838d7fc0dba0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120942025-04-30T06:42:51.708ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
120952025-04-30T06:42:51.711ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f547a9d6-7c33-44d1-abf6-6d04c03557cc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
120962025-04-30T06:42:51.711ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
120972025-04-30T06:42:51.714ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120982025-04-30T06:42:51.714ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120992025-04-30T06:42:51.714ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121002025-04-30T06:42:51.715ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
121012025-04-30T06:42:51.717ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121022025-04-30T06:42:51.717ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "468e4529-8038-4a84-8df2-dc5f82d2d73e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121032025-04-30T06:42:51.717ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
121042025-04-30T06:42:51.718ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "269ec26e-8903-4282-8450-66010f605f18", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121052025-04-30T06:42:51.718ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
121062025-04-30T06:42:51.724ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121072025-04-30T06:42:51.741ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121082025-04-30T06:42:51.741ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121092025-04-30T06:42:51.741ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
121102025-04-30T06:42:51.741ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121112025-04-30T06:42:51.743ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "093aba35-80ec-4954-8802-2e8759ecb2bf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121122025-04-30T06:42:51.743ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
121132025-04-30T06:42:51.745ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52149de4-2426-4167-b176-501ded7aea9d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121142025-04-30T06:42:51.745ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
121152025-04-30T06:42:51.747ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121162025-04-30T06:42:51.747ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
121172025-04-30T06:42:51.748ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121182025-04-30T06:42:51.751ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87ca853a-29d0-432b-876f-05a96f25613b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121192025-04-30T06:42:51.751ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
121202025-04-30T06:42:51.754ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e11a489-5cad-4c2c-9e20-338aecbcfc47", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121212025-04-30T06:42:51.754ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
121222025-04-30T06:42:51.757ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "672fdb16-c371-46dd-b62e-5e51bf34b5b3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121232025-04-30T06:42:51.757ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
121242025-04-30T06:42:51.762ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7af0507-4bed-44c7-b2fb-9c6c5b72a65b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121252025-04-30T06:42:51.762ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
121262025-04-30T06:42:51.766ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79c9bfba-dc40-4ffc-9e05-fbf760e2529b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121272025-04-30T06:42:51.766ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
121282025-04-30T06:42:51.768ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de1a7b70-7c69-4164-8434-89bbfae1badb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121292025-04-30T06:42:51.768ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
121302025-04-30T06:42:51.769ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6270d085-9e61-4ea4-9a08-5cfb454a7458", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121312025-04-30T06:42:51.770ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
121322025-04-30T06:42:51.771ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "448bb0bf-5a94-4de0-b415-a6b56d81f02f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121332025-04-30T06:42:51.771ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
121342025-04-30T06:42:51.772ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1de6df26-ed32-44be-bf6a-2a3cfb495dd9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121352025-04-30T06:42:51.773ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
121362025-04-30T06:42:51.774ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3698ee6-7343-4260-bc70-256903f26d78", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121372025-04-30T06:42:51.774ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
121382025-04-30T06:42:51.776ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01b345e4-a854-498f-a981-bb545e4c7c6f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121392025-04-30T06:42:51.776ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
121402025-04-30T06:42:51.779ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
121412025-04-30T06:42:51.782ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e51deb4f-f749-43d1-8797-821593fa2259", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121422025-04-30T06:42:51.782ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
121432025-04-30T06:42:51.785ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "641a2b36-1c83-44bb-9f94-d9515ab9d455", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121442025-04-30T06:42:51.785ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
121452025-04-30T06:42:51.787ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7e9757-da93-4250-afed-db3bf40170c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121462025-04-30T06:42:51.787ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
121472025-04-30T06:42:51.799ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fcfd251-7f22-4b9b-9264-6f416fe0c1f3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121482025-04-30T06:42:51.799ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
121492025-04-30T06:42:51.802ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62d207a2-4172-4974-9ac6-84d5ae816fd4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121502025-04-30T06:42:51.802ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
121512025-04-30T06:42:51.807ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcb20f41-776b-49cd-bfe6-001eccf53e80", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121522025-04-30T06:42:51.807ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
121532025-04-30T06:42:51.808ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18e9f1ff-cafe-4556-90ee-58fd18aab50e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121542025-04-30T06:42:51.808ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
121552025-04-30T06:42:51.809ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14b66bfe-f826-49bf-a3d0-f71612ab9ae7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121562025-04-30T06:42:51.809ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
121572025-04-30T06:42:51.811ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b17b79d-8713-4fd6-b1be-bc20dafb8366", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121582025-04-30T06:42:51.811ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
121592025-04-30T06:42:51.816ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "830dc92d-3490-4962-801f-dfe15a1eb609", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
121602025-04-30T06:42:51.816ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
121612025-04-30T06:42:51.816ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121622025-04-30T06:42:51.816ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121632025-04-30T06:42:51.817ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121642025-04-30T06:42:51.817ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121652025-04-30T06:42:51.824ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121662025-04-30T06:42:51.833ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
121672025-04-30T06:42:51.833ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
121682025-04-30T06:42:51.833ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121692025-04-30T06:42:51.833ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121702025-04-30T06:42:51.833ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121712025-04-30T06:42:51.834ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121722025-04-30T06:42:51.835ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121732025-04-30T06:42:51.836ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121742025-04-30T06:42:51.837ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121752025-04-30T06:42:51.837ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
121762025-04-30T06:42:51.837ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121772025-04-30T06:42:51.845ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121782025-04-30T06:42:51.850ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
121792025-04-30T06:42:51.850ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121802025-04-30T06:42:51.850ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121812025-04-30T06:42:51.851ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121822025-04-30T06:42:51.853ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121832025-04-30T06:42:51.858ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121842025-04-30T06:42:51.859ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121852025-04-30T06:42:51.859ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
121862025-04-30T06:42:51.859ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121872025-04-30T06:42:51.863ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121882025-04-30T06:42:51.863ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
121892025-04-30T06:42:51.864ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121902025-04-30T06:42:51.944ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
121912025-04-30T06:42:51.944ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
121922025-04-30T06:42:51.945ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:46015 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
121932025-04-30T06:42:51.947ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
121942025-04-30T06:42:51.947ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
121952025-04-30T06:42:51.947ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
121962025-04-30T06:42:51.948ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
121972025-04-30T06:42:51.949ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:46015 req_id = be074042-d556-47f9-9fc8-1168cbcbb3da sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
121982025-04-30T06:42:51.949ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 416 local_addr = [::1]:62607 method = GET remote_addr = [::1]:46015 req_id = be074042-d556-47f9-9fc8-1168cbcbb3da response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
121992025-04-30T06:42:51.949ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "be074042-d556-47f9-9fc8-1168cbcbb3da", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
122002025-04-30T06:42:51.952ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122012025-04-30T06:42:51.952ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122022025-04-30T06:42:51.953ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
122032025-04-30T06:42:51.959ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122042025-04-30T06:42:51.959ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122052025-04-30T06:42:51.959ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122062025-04-30T06:42:51.959ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122072025-04-30T06:42:51.959ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122082025-04-30T06:42:51.966ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
122092025-04-30T06:42:51.966ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
122102025-04-30T06:42:51.967ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122112025-04-30T06:42:51.972ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122122025-04-30T06:42:51.972ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122132025-04-30T06:42:51.972ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122142025-04-30T06:42:51.973ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122152025-04-30T06:42:51.977ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
122162025-04-30T06:42:51.977ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122172025-04-30T06:42:52.010ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
122182025-04-30T06:42:52.026ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
122192025-04-30T06:42:52.028ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
122202025-04-30T06:42:52.028ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 490.135813ms iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
122212025-04-30T06:42:52.028ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
122222025-04-30T06:42:52.028ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122232025-04-30T06:42:52.028ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122242025-04-30T06:42:52.028ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122252025-04-30T06:42:52.028ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122262025-04-30T06:42:52.029ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122272025-04-30T06:42:52.033ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122282025-04-30T06:42:52.037ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122292025-04-30T06:42:52.037ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122302025-04-30T06:42:52.037ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122312025-04-30T06:42:52.038ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122322025-04-30T06:42:52.042ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
122332025-04-30T06:42:52.042ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
122342025-04-30T06:42:52.043ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c1 a3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63275 req_id = 8afc7e57-1114-4b2e-a21d-2c0827cf8770
122352025-04-30T06:42:52.043ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49571,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63275 req_id = 8afc7e57-1114-4b2e-a21d-2c0827cf8770
122362025-04-30T06:42:52.044ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
122372025-04-30T06:42:52.044ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122382025-04-30T06:42:52.044ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63275 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8afc7e57-1114-4b2e-a21d-2c0827cf8770
122392025-04-30T06:42:52.044ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4236, tv_nsec: 843520649 } })
122402025-04-30T06:42:52.044ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
122412025-04-30T06:42:52.044ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 07 7d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .}..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55567 req_id = e958e9ee-4b2e-4186-ab1f-c3e56efa9122
122422025-04-30T06:42:52.045ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1917,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55567 req_id = e958e9ee-4b2e-4186-ab1f-c3e56efa9122
122432025-04-30T06:42:52.045ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122442025-04-30T06:42:52.045ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55567 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e958e9ee-4b2e-4186-ab1f-c3e56efa9122
122452025-04-30T06:42:52.111ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
122462025-04-30T06:42:52.112ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122472025-04-30T06:42:52.112ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122482025-04-30T06:42:52.112ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122492025-04-30T06:42:52.112ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
122502025-04-30T06:42:52.113ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122512025-04-30T06:42:52.118ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
122522025-04-30T06:42:52.119ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
122532025-04-30T06:42:52.123ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7d f8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c }............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57663 req_id = cdc99185-0865-4176-8188-b17f3cff3da5
122542025-04-30T06:42:52.123ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32248,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57663 req_id = cdc99185-0865-4176-8188-b17f3cff3da5
122552025-04-30T06:42:52.123ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
122562025-04-30T06:42:52.123ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32248, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57663 req_id = cdc99185-0865-4176-8188-b17f3cff3da5
122572025-04-30T06:42:52.125ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
122582025-04-30T06:42:52.126ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
122592025-04-30T06:42:52.126ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
122602025-04-30T06:42:52.128ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 5c 10 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 \\............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:50035 req_id = 9f5e9f29-e242-443f-a363-b3b64af3157c
122612025-04-30T06:42:52.128ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23568,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50035 req_id = 9f5e9f29-e242-443f-a363-b3b64af3157c
122622025-04-30T06:42:52.128ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
122632025-04-30T06:42:52.128ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122642025-04-30T06:42:52.129ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50035 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9f5e9f29-e242-443f-a363-b3b64af3157c
122652025-04-30T06:42:52.130ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122662025-04-30T06:42:52.138ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122672025-04-30T06:42:52.139ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122682025-04-30T06:42:52.139ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
122692025-04-30T06:42:52.140ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122702025-04-30T06:42:52.142ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4236, tv_nsec: 941251100 } })
122712025-04-30T06:42:52.143ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
122722025-04-30T06:42:52.149ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 76 64 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e vd..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51876 req_id = c9274876-46fd-4eb9-973f-a4af5b760e9b
122732025-04-30T06:42:52.149ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30308,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51876 req_id = c9274876-46fd-4eb9-973f-a4af5b760e9b
122742025-04-30T06:42:52.149ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122752025-04-30T06:42:52.149ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51876 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c9274876-46fd-4eb9-973f-a4af5b760e9b
122762025-04-30T06:42:52.150ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
122772025-04-30T06:42:52.202ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
122782025-04-30T06:42:52.202ZDEBGtest_delete_snapshot (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
122792025-04-30T06:42:52.202ZDEBGtest_delete_snapshot (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
122802025-04-30T06:42:52.202ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
122812025-04-30T06:42:52.203ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
122822025-04-30T06:42:52.209ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122832025-04-30T06:42:52.213ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
122842025-04-30T06:42:52.213ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122852025-04-30T06:42:52.214ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
122862025-04-30T06:42:52.228ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 74 7b 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 t{..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52899 req_id = 2c36978a-80f1-4000-a89e-9bf27f41811e
122872025-04-30T06:42:52.229ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29819,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52899 req_id = 2c36978a-80f1-4000-a89e-9bf27f41811e
122882025-04-30T06:42:52.229ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
122892025-04-30T06:42:52.229ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122902025-04-30T06:42:52.229ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52899 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2c36978a-80f1-4000-a89e-9bf27f41811e
122912025-04-30T06:42:52.230ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9aebdd0c-b663-4624-9e2f-31dc16b15d2b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
122922025-04-30T06:42:52.232ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
122932025-04-30T06:42:52.233ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4237, tv_nsec: 32344057 } })
122942025-04-30T06:42:52.233ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
122952025-04-30T06:42:52.252ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ee a4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56678 req_id = 902e5066-77f8-459a-aa40-794e84937526
122962025-04-30T06:42:52.252ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61092,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56678 req_id = 902e5066-77f8-459a-aa40-794e84937526
122972025-04-30T06:42:52.252ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122982025-04-30T06:42:52.252ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56678 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 902e5066-77f8-459a-aa40-794e84937526
122992025-04-30T06:42:52.253ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
123002025-04-30T06:42:52.326ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
123012025-04-30T06:42:52.328ZDEBGtest_delete_snapshot (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
123022025-04-30T06:42:52.328ZDEBGtest_delete_snapshot (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
123032025-04-30T06:42:52.328ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
123042025-04-30T06:42:52.331ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
123052025-04-30T06:42:52.332ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
123062025-04-30T06:42:52.341ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8a1d24e-06c9-4d39-b918-0c4cec62577c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123072025-04-30T06:42:52.341ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
123082025-04-30T06:42:52.343ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 28 37 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 (7..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53248 req_id = 410776a5-ea2b-47b4-a40c-eaf552fcaa6c
123092025-04-30T06:42:52.344ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10295,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53248 req_id = 410776a5-ea2b-47b4-a40c-eaf552fcaa6c
123102025-04-30T06:42:52.345ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
123112025-04-30T06:42:52.345ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123122025-04-30T06:42:52.345ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53248 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 410776a5-ea2b-47b4-a40c-eaf552fcaa6c
123132025-04-30T06:42:52.352ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4237, tv_nsec: 151202194 } })
123142025-04-30T06:42:52.352ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
123152025-04-30T06:42:52.356ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2b 7b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e +{..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55958 req_id = f6c6432e-9da0-42ae-a7ba-b74fee11bb2c
123162025-04-30T06:42:52.356ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11131,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55958 req_id = f6c6432e-9da0-42ae-a7ba-b74fee11bb2c
123172025-04-30T06:42:52.356ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123182025-04-30T06:42:52.356ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55958 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f6c6432e-9da0-42ae-a7ba-b74fee11bb2c
123192025-04-30T06:42:52.357ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
123202025-04-30T06:42:52.419ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
123212025-04-30T06:42:52.419ZDEBGtest_delete_snapshot (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
123222025-04-30T06:42:52.419ZDEBGtest_delete_snapshot (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
123232025-04-30T06:42:52.419ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
123242025-04-30T06:42:52.421ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
123252025-04-30T06:42:52.423ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bf3d1f8-a5ea-4adb-b385-452a732cb980", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123262025-04-30T06:42:52.423ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
123272025-04-30T06:42:52.426ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e140f728-09ee-49fb-be55-e52b29658f74", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123282025-04-30T06:42:52.426ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
123292025-04-30T06:42:52.429ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13285913-7996-45b8-a70a-df3a81095194", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123302025-04-30T06:42:52.429ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
123312025-04-30T06:42:52.435ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d312de56-2650-4209-b729-3ad88467dc06", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123322025-04-30T06:42:52.435ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
123332025-04-30T06:42:52.435ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
123342025-04-30T06:42:52.438ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4392f3fc-dfd5-40e6-b121-060ad34c976c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123352025-04-30T06:42:52.438ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
123362025-04-30T06:42:52.440ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e37c2711-dabd-4645-b4e4-4a938ad04c75", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123372025-04-30T06:42:52.440ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
123382025-04-30T06:42:52.441ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03bb8bf5-186c-4473-993e-7c4821f23306", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123392025-04-30T06:42:52.441ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
123402025-04-30T06:42:52.442ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40f97a33-8d76-4c52-b5f9-331b03dd941a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123412025-04-30T06:42:52.443ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
123422025-04-30T06:42:52.444ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64ed3963-073f-4722-84f5-59433ffd3a38", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123432025-04-30T06:42:52.444ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
123442025-04-30T06:42:52.445ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ca820d-4521-4bb0-9391-a5d309c6c2ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123452025-04-30T06:42:52.445ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
123462025-04-30T06:42:52.446ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c31227b5-8885-49d7-8a87-813ffcf72472", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123472025-04-30T06:42:52.446ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
123482025-04-30T06:42:52.450ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0783d59b-2b25-4bb6-aca4-eab18c0116a9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123492025-04-30T06:42:52.450ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
123502025-04-30T06:42:52.452ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b710b8ca-a5f9-403c-9888-a3b157b148c6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:51 GMT"} })
123512025-04-30T06:42:52.452ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
123522025-04-30T06:42:52.454ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2d99d82-a63e-4316-96fa-57fdad621bba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123532025-04-30T06:42:52.454ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
123542025-04-30T06:42:52.457ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d51f1ef0-f539-4fe7-8b91-95ebd67eee3f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123552025-04-30T06:42:52.458ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
123562025-04-30T06:42:52.460ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fba83b4-8042-46d5-ac48-f0f9ed96000b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123572025-04-30T06:42:52.460ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
123582025-04-30T06:42:52.462ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a05109e4-3f44-474e-b517-e2e4a96236c1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123592025-04-30T06:42:52.462ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
123602025-04-30T06:42:52.463ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "541f2692-05a9-43f7-bb6d-797ee4df6292", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123612025-04-30T06:42:52.463ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
123622025-04-30T06:42:52.464ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "412df040-6546-4f79-adce-cb6a236b2669", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123632025-04-30T06:42:52.464ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
123642025-04-30T06:42:52.465ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e6b04c-bbcf-4b9b-9ee3-a93c6640d580", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123652025-04-30T06:42:52.465ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
123662025-04-30T06:42:52.466ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83c11eeb-6464-4d60-a6b7-8b6f88bfa617", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123672025-04-30T06:42:52.467ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
123682025-04-30T06:42:52.468ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a158556-e5f5-4fe7-8974-1e692d674d10", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123692025-04-30T06:42:52.468ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
123702025-04-30T06:42:52.470ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "adc62cb3-6674-46f8-b0c1-69598632d2ea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123712025-04-30T06:42:52.470ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
123722025-04-30T06:42:52.474ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c5b57a2-b168-4194-8cdc-978493410d46", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123732025-04-30T06:42:52.474ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
123742025-04-30T06:42:52.477ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ea32e9b-e9e5-483e-bf2f-e4b7841aca50", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123752025-04-30T06:42:52.477ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
123762025-04-30T06:42:52.478ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75312a21-5482-4d11-a0d4-65638e1d0819", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123772025-04-30T06:42:52.478ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
123782025-04-30T06:42:52.479ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "161d694d-9099-4495-9eda-64534c7bf7d6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123792025-04-30T06:42:52.480ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
123802025-04-30T06:42:52.482ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d9e11e2-9135-4f06-a361-476344dc0f15", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123812025-04-30T06:42:52.482ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
123822025-04-30T06:42:52.483ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7ecb512-33c5-447b-89ac-2c51555dd946", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123832025-04-30T06:42:52.483ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
123842025-04-30T06:42:52.492ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123852025-04-30T06:42:52.492ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123862025-04-30T06:42:52.492ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123872025-04-30T06:42:52.492ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
123882025-04-30T06:42:52.492ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123892025-04-30T06:42:52.494ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc811489-dfa0-43b5-a6ab-b3a3323fc56a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123902025-04-30T06:42:52.494ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
123912025-04-30T06:42:52.495ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123922025-04-30T06:42:52.506ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123932025-04-30T06:42:52.506ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123942025-04-30T06:42:52.506ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
123952025-04-30T06:42:52.508ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123962025-04-30T06:42:52.510ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcfcf453-4465-41d8-8919-33b66f28fae4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123972025-04-30T06:42:52.510ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
123982025-04-30T06:42:52.512ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2431019c-14a4-4627-964a-aa8fbe5c8260", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
123992025-04-30T06:42:52.512ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
124002025-04-30T06:42:52.512ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124012025-04-30T06:42:52.512ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
124022025-04-30T06:42:52.513ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124032025-04-30T06:42:52.515ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a3d2388-5975-4ca1-ae56-5385a88f3962", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124042025-04-30T06:42:52.516ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
124052025-04-30T06:42:52.519ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fcb85ec-373d-4a93-97c7-bed15d5e4e38", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124062025-04-30T06:42:52.519ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
124072025-04-30T06:42:52.522ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9e63e90-776e-489f-93bf-4075b2c4ca81", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124082025-04-30T06:42:52.522ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
124092025-04-30T06:42:52.524ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6a6e63e-f0c7-4b41-b862-bccbadcb6162", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124102025-04-30T06:42:52.524ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
124112025-04-30T06:42:52.527ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49593158-a8d8-46b2-9c82-cbaf6aab5fa6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124122025-04-30T06:42:52.528ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
124132025-04-30T06:42:52.531ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84973958-14a8-4669-940f-729e4b501da8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124142025-04-30T06:42:52.531ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
124152025-04-30T06:42:52.535ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8b05174-1cd1-44a3-8044-10b3c37cc934", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124162025-04-30T06:42:52.535ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
124172025-04-30T06:42:52.536ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3172048b-4e02-4afb-9eff-d51bca092e45", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124182025-04-30T06:42:52.537ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
124192025-04-30T06:42:52.538ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "931f0c37-b0f6-419c-996b-304b6113ebc2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124202025-04-30T06:42:52.539ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
124212025-04-30T06:42:52.548ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f54097d4-2e60-4206-96eb-4569af1a01c3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124222025-04-30T06:42:52.548ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
124232025-04-30T06:42:52.549ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
124242025-04-30T06:42:52.551ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b289bd96-9983-4665-b875-babaa94cb36b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124252025-04-30T06:42:52.551ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124262025-04-30T06:42:52.551ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124272025-04-30T06:42:52.551ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124282025-04-30T06:42:52.551ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124292025-04-30T06:42:52.551ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124302025-04-30T06:42:52.552ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124312025-04-30T06:42:52.552ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124322025-04-30T06:42:52.554ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124332025-04-30T06:42:52.555ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124342025-04-30T06:42:52.555ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124352025-04-30T06:42:52.555ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124362025-04-30T06:42:52.556ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124372025-04-30T06:42:52.563ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124382025-04-30T06:42:52.563ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124392025-04-30T06:42:52.563ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124402025-04-30T06:42:52.563ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124412025-04-30T06:42:52.563ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124422025-04-30T06:42:52.564ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124432025-04-30T06:42:52.571ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124442025-04-30T06:42:52.582ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124452025-04-30T06:42:52.582ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124462025-04-30T06:42:52.582ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124472025-04-30T06:42:52.584ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124482025-04-30T06:42:52.610ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124492025-04-30T06:42:52.611ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124502025-04-30T06:42:52.611ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124512025-04-30T06:42:52.611ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
124522025-04-30T06:42:52.623ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124532025-04-30T06:42:52.626ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124542025-04-30T06:42:52.657ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124552025-04-30T06:42:52.657ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124562025-04-30T06:42:52.657ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
124572025-04-30T06:42:52.658ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124582025-04-30T06:42:52.761ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
124592025-04-30T06:42:52.762ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
124602025-04-30T06:42:52.764ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:41899 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
124612025-04-30T06:42:52.768ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124622025-04-30T06:42:52.768ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
124632025-04-30T06:42:52.769ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124642025-04-30T06:42:52.769ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:41899 req_id = 2f7bbdee-52bc-4060-8c81-7ead61fd3e06 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124652025-04-30T06:42:52.770ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 753 local_addr = [::1]:62607 method = GET remote_addr = [::1]:41899 req_id = 2f7bbdee-52bc-4060-8c81-7ead61fd3e06 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124662025-04-30T06:42:52.771ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f7bbdee-52bc-4060-8c81-7ead61fd3e06", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
124672025-04-30T06:42:52.775ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124682025-04-30T06:42:52.775ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124692025-04-30T06:42:52.775ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
124702025-04-30T06:42:52.777ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124712025-04-30T06:42:52.777ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124722025-04-30T06:42:52.777ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124732025-04-30T06:42:52.777ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124742025-04-30T06:42:52.779ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124752025-04-30T06:42:52.790ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124762025-04-30T06:42:52.795ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124772025-04-30T06:42:52.795ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124782025-04-30T06:42:52.795ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124792025-04-30T06:42:52.796ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124802025-04-30T06:42:52.899ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
124812025-04-30T06:42:52.904ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124822025-04-30T06:42:52.904ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124832025-04-30T06:42:52.905ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
124842025-04-30T06:42:52.908ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
124852025-04-30T06:42:52.915ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
124862025-04-30T06:42:52.915ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
124872025-04-30T06:42:52.920ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
124882025-04-30T06:42:52.920ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124892025-04-30T06:42:52.921ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
124902025-04-30T06:42:52.922ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
124912025-04-30T06:42:52.922ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 893.824167ms iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
124922025-04-30T06:42:52.922ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
124932025-04-30T06:42:52.922ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124942025-04-30T06:42:52.927ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124952025-04-30T06:42:52.934ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124962025-04-30T06:42:52.934ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124972025-04-30T06:42:52.943ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124982025-04-30T06:42:52.949ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124992025-04-30T06:42:52.971ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
125002025-04-30T06:42:52.971ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125012025-04-30T06:42:52.971ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125022025-04-30T06:42:52.972ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125032025-04-30T06:42:52.982ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125042025-04-30T06:42:53.007ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125052025-04-30T06:42:53.007ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125062025-04-30T06:42:53.007ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
125072025-04-30T06:42:53.009ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125082025-04-30T06:42:53.016ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
125092025-04-30T06:42:53.031ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125102025-04-30T06:42:53.037ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125112025-04-30T06:42:53.071ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125122025-04-30T06:42:53.071ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
125132025-04-30T06:42:53.072ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125142025-04-30T06:42:53.074ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
125152025-04-30T06:42:53.074ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
125162025-04-30T06:42:53.079ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1f 84 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52011 req_id = 1604c604-76b9-4cc8-876a-ac49358fe527
125172025-04-30T06:42:53.080ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8068,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52011 req_id = 1604c604-76b9-4cc8-876a-ac49358fe527
125182025-04-30T06:42:53.080ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
125192025-04-30T06:42:53.080ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125202025-04-30T06:42:53.080ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52011 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1604c604-76b9-4cc8-876a-ac49358fe527
125212025-04-30T06:42:53.080ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125222025-04-30T06:42:53.080ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
125232025-04-30T06:42:53.081ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125242025-04-30T06:42:53.082ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4237, tv_nsec: 881593431 } })
125252025-04-30T06:42:53.082ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
125262025-04-30T06:42:53.097ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 36 a3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 6...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59876 req_id = c10e46f1-9772-442d-a985-f067ecb15dd3
125272025-04-30T06:42:53.097ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13987,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59876 req_id = c10e46f1-9772-442d-a985-f067ecb15dd3
125282025-04-30T06:42:53.098ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125292025-04-30T06:42:53.098ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59876 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c10e46f1-9772-442d-a985-f067ecb15dd3
125302025-04-30T06:42:53.202ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
125312025-04-30T06:42:53.212ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 30 06 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 0............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49668 req_id = 65c38488-4494-4d8c-8a49-a75ce5386ec5
125322025-04-30T06:42:53.212ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12294,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49668 req_id = 65c38488-4494-4d8c-8a49-a75ce5386ec5
125332025-04-30T06:42:53.212ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
125342025-04-30T06:42:53.212ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12294, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49668 req_id = 65c38488-4494-4d8c-8a49-a75ce5386ec5
125352025-04-30T06:42:53.213ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
125362025-04-30T06:42:53.214ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
125372025-04-30T06:42:53.214ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
125382025-04-30T06:42:53.231ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5173f891-46b2-4dfd-883c-a3e6af1a2663", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125392025-04-30T06:42:53.231ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
125402025-04-30T06:42:53.259ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1d950b2-ce76-4e45-8de2-d289cd7ff20e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125412025-04-30T06:42:53.260ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
125422025-04-30T06:42:53.270ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfc667f0-4af9-4878-a9b5-8f65bdb14518", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125432025-04-30T06:42:53.270ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
125442025-04-30T06:42:53.272ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25029120-6838-4fa1-9d65-d7b8fab457c8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125452025-04-30T06:42:53.272ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
125462025-04-30T06:42:53.281ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c64ce9c0-52b7-48f5-810c-ee62822e59da", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125472025-04-30T06:42:53.285ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
125482025-04-30T06:42:53.292ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ddf7da4-1a5e-459d-a7f6-4b02461fc041", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125492025-04-30T06:42:53.292ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
125502025-04-30T06:42:53.303ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
125512025-04-30T06:42:53.313ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36616a63-6098-4d0a-bd26-189df084e372", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125522025-04-30T06:42:53.313ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
125532025-04-30T06:42:53.316ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07ec4419-d77f-47a7-a5e1-56d5ec048d10", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125542025-04-30T06:42:53.316ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
125552025-04-30T06:42:53.317ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96ce0cfd-88dc-4e07-9053-312e30b1a165", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125562025-04-30T06:42:53.317ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
125572025-04-30T06:42:53.318ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ef893ae-0826-4d4e-9615-0571473aee21", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125582025-04-30T06:42:53.318ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
125592025-04-30T06:42:53.319ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de68b1c6-6bf4-4ddd-9ed6-11ef71692c9d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125602025-04-30T06:42:53.319ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
125612025-04-30T06:42:53.320ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37c93fa7-2de8-4985-8401-4c76971fb11a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125622025-04-30T06:42:53.320ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
125632025-04-30T06:42:53.324ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4703037a-2b2b-4ad3-8e95-08ca5641e5bb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125642025-04-30T06:42:53.324ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
125652025-04-30T06:42:53.326ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6da7a32c-4058-41c4-8553-998bb325a044", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125662025-04-30T06:42:53.326ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
125672025-04-30T06:42:53.328ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bce13ea-2e70-4adc-9e0e-1efcca30bf19", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125682025-04-30T06:42:53.329ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
125692025-04-30T06:42:53.331ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd7c8ac1-0517-4459-ac54-34010a819526", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125702025-04-30T06:42:53.332ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
125712025-04-30T06:42:53.334ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e32024f-5585-4923-8581-d9c86d16747e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125722025-04-30T06:42:53.334ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
125732025-04-30T06:42:53.336ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9d27ba3-445e-4b63-9d8d-d992ecbf5b0c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125742025-04-30T06:42:53.336ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
125752025-04-30T06:42:53.338ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92e83a55-b998-4546-80ab-096da47fad61", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125762025-04-30T06:42:53.339ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
125772025-04-30T06:42:53.340ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e6f4abf-cd9a-4ce6-87f0-31ae1f4899b5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125782025-04-30T06:42:53.340ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
125792025-04-30T06:42:53.341ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5843ffe-cdf9-4d6f-b72d-ea0845b6a02a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125802025-04-30T06:42:53.341ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
125812025-04-30T06:42:53.342ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ead7c4e1-9b36-4922-b27d-487a4c8c62e7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125822025-04-30T06:42:53.343ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
125832025-04-30T06:42:53.344ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a129261-0b3a-4f5b-81dc-9b600aff7899", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125842025-04-30T06:42:53.344ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
125852025-04-30T06:42:53.345ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e91babbf-ced2-4162-bae0-d1ea94c2853b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125862025-04-30T06:42:53.345ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
125872025-04-30T06:42:53.350ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cf81592-87bf-48d0-9a0b-2fbfd5787e62", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125882025-04-30T06:42:53.350ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
125892025-04-30T06:42:53.353ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca02e2d1-5b80-4beb-a01b-a4163979953d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125902025-04-30T06:42:53.353ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
125912025-04-30T06:42:53.356ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125922025-04-30T06:42:53.356ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125932025-04-30T06:42:53.356ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125942025-04-30T06:42:53.357ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
125952025-04-30T06:42:53.357ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125962025-04-30T06:42:53.359ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e2a0f80-369a-4dc9-8cca-49f9241d6213", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
125972025-04-30T06:42:53.359ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
125982025-04-30T06:42:53.360ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125992025-04-30T06:42:53.369ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126002025-04-30T06:42:53.369ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126012025-04-30T06:42:53.369ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
126022025-04-30T06:42:53.370ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126032025-04-30T06:42:53.373ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126042025-04-30T06:42:53.373ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
126052025-04-30T06:42:53.375ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126062025-04-30T06:42:53.376ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1fe8d5a-a91c-4319-a97b-41e3e28ad4c2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126072025-04-30T06:42:53.376ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
126082025-04-30T06:42:53.382ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01dd9c4d-f318-4f24-a52b-5637ff2ad16f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126092025-04-30T06:42:53.382ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
126102025-04-30T06:42:53.384ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ca7077a-5d9d-401c-999c-7d3361d05afb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126112025-04-30T06:42:53.384ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
126122025-04-30T06:42:53.385ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe330a4e-0d6d-4698-b759-f11d7086c665", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126132025-04-30T06:42:53.386ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
126142025-04-30T06:42:53.387ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "feca8739-cb18-4574-8b9b-e7e820864892", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126152025-04-30T06:42:53.387ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
126162025-04-30T06:42:53.388ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fbe5629-2951-4d78-b4ca-aee3f42fe325", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126172025-04-30T06:42:53.388ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
126182025-04-30T06:42:53.392ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8e60907-d511-4a0a-a7b9-63c0c0b5c02a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126192025-04-30T06:42:53.393ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
126202025-04-30T06:42:53.396ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc35c4ec-85f8-40fc-a88f-e7acc84855be", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126212025-04-30T06:42:53.396ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
126222025-04-30T06:42:53.399ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92fc063b-8737-4f2c-944f-d0bbaeee83ac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126232025-04-30T06:42:53.399ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
126242025-04-30T06:42:53.401ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc376318-5405-4513-91c8-46adffcf38d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126252025-04-30T06:42:53.401ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
126262025-04-30T06:42:53.405ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f0cd1ca-e579-4064-b246-cccb9a78fc33", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126272025-04-30T06:42:53.405ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
126282025-04-30T06:42:53.408ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcb002e5-7b8b-4b5b-a22a-47d652b19097", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126292025-04-30T06:42:53.408ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
126302025-04-30T06:42:53.411ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0797f47-9c95-4051-b82e-69c4d6ce2f76", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126312025-04-30T06:42:53.411ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
126322025-04-30T06:42:53.416ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "127a0844-f64f-4c4c-9d66-601333748e5e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126332025-04-30T06:42:53.423ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
126342025-04-30T06:42:53.429ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
126352025-04-30T06:42:53.432ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e2b972-2549-4a82-85ac-d8e86b57c265", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126362025-04-30T06:42:53.433ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
126372025-04-30T06:42:53.437ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79bee371-de88-4631-b976-f82783ac1fb4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126382025-04-30T06:42:53.437ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
126392025-04-30T06:42:53.439ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b717377-46bc-4510-869c-7ef27a5cc16a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126402025-04-30T06:42:53.439ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
126412025-04-30T06:42:53.442ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "288b0d7a-f6b3-44b5-ace6-e9c4f024c912", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:52 GMT"} })
126422025-04-30T06:42:53.442ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126432025-04-30T06:42:53.442ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126442025-04-30T06:42:53.442ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126452025-04-30T06:42:53.442ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126462025-04-30T06:42:53.443ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126472025-04-30T06:42:53.471ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126482025-04-30T06:42:53.474ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126492025-04-30T06:42:53.474ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126502025-04-30T06:42:53.474ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126512025-04-30T06:42:53.474ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126522025-04-30T06:42:53.474ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126532025-04-30T06:42:53.475ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126542025-04-30T06:42:53.494ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126552025-04-30T06:42:53.494ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126562025-04-30T06:42:53.494ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126572025-04-30T06:42:53.494ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
126582025-04-30T06:42:53.494ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126592025-04-30T06:42:53.494ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
126602025-04-30T06:42:53.495ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126612025-04-30T06:42:53.495ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126622025-04-30T06:42:53.503ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126632025-04-30T06:42:53.503ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126642025-04-30T06:42:53.503ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126652025-04-30T06:42:53.505ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126662025-04-30T06:42:53.531ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3009bb36-0284-4900-a774-f6bd3f5e03c6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126672025-04-30T06:42:53.532ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3009bb36-0284-4900-a774-f6bd3f5e03c6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126682025-04-30T06:42:53.532ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3009bb36-0284-4900-a774-f6bd3f5e03c6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126692025-04-30T06:42:53.534ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3009bb36-0284-4900-a774-f6bd3f5e03c6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126702025-04-30T06:42:53.536ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126712025-04-30T06:42:53.557ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126722025-04-30T06:42:53.557ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126732025-04-30T06:42:53.557ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
126742025-04-30T06:42:53.558ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126752025-04-30T06:42:53.711ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
126762025-04-30T06:42:53.711ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
126772025-04-30T06:42:53.712ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 181178 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3009bb36-0284-4900-a774-f6bd3f5e03c6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126782025-04-30T06:42:53.713ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:59503 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
126792025-04-30T06:42:53.713ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126802025-04-30T06:42:53.714ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "3009bb36-0284-4900-a774-f6bd3f5e03c6", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
126812025-04-30T06:42:53.714ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
126822025-04-30T06:42:53.714ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126832025-04-30T06:42:53.714ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3e830568-1314-4bc6-90f5-697201954cc1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
126842025-04-30T06:42:53.714ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:59503 req_id = 249f8fe9-04f9-4d3d-b254-c16065e8416b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126852025-04-30T06:42:53.716ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3e830568-1314-4bc6-90f5-697201954cc1 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
126862025-04-30T06:42:53.716ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3e830568-1314-4bc6-90f5-697201954cc1 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
126872025-04-30T06:42:53.718ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3e830568-1314-4bc6-90f5-697201954cc1 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
126882025-04-30T06:42:53.721ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 6506 local_addr = [::1]:62607 method = GET remote_addr = [::1]:59503 req_id = 249f8fe9-04f9-4d3d-b254-c16065e8416b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126892025-04-30T06:42:53.722ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126902025-04-30T06:42:53.722ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
126912025-04-30T06:42:53.723ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126922025-04-30T06:42:53.723ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "249f8fe9-04f9-4d3d-b254-c16065e8416b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
126932025-04-30T06:42:53.727ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126942025-04-30T06:42:53.727ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126952025-04-30T06:42:53.727ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
126962025-04-30T06:42:53.728ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126972025-04-30T06:42:53.728ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126982025-04-30T06:42:53.728ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126992025-04-30T06:42:53.729ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127002025-04-30T06:42:53.729ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127012025-04-30T06:42:53.729ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127022025-04-30T06:42:53.731ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127032025-04-30T06:42:53.731ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127042025-04-30T06:42:53.731ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127052025-04-30T06:42:53.733ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 18937 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 3e830568-1314-4bc6-90f5-697201954cc1 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
127062025-04-30T06:42:53.737ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e830568-1314-4bc6-90f5-697201954cc1", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127072025-04-30T06:42:53.737ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127082025-04-30T06:42:53.737ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
127092025-04-30T06:42:53.737ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
127102025-04-30T06:42:53.738ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127112025-04-30T06:42:53.742ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
127122025-04-30T06:42:53.742ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127132025-04-30T06:42:53.742ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127142025-04-30T06:42:53.743ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127152025-04-30T06:42:53.865ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
127162025-04-30T06:42:53.866ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
127172025-04-30T06:42:53.866ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 943.541614ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
127182025-04-30T06:42:53.866ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
127192025-04-30T06:42:53.866ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
127202025-04-30T06:42:53.866ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127212025-04-30T06:42:53.866ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127222025-04-30T06:42:53.866ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127232025-04-30T06:42:53.867ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127242025-04-30T06:42:53.870ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
127252025-04-30T06:42:53.871ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127262025-04-30T06:42:53.874ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
127272025-04-30T06:42:53.874ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127282025-04-30T06:42:53.874ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127292025-04-30T06:42:53.875ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127302025-04-30T06:42:53.883ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
127312025-04-30T06:42:53.884ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
127322025-04-30T06:42:53.885ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cf a7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60845 req_id = bc48cddc-cc5b-4f8c-9e6b-88d52377a618
127332025-04-30T06:42:53.885ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53159,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60845 req_id = bc48cddc-cc5b-4f8c-9e6b-88d52377a618
127342025-04-30T06:42:53.886ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
127352025-04-30T06:42:53.886ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127362025-04-30T06:42:53.886ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60845 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bc48cddc-cc5b-4f8c-9e6b-88d52377a618
127372025-04-30T06:42:53.886ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4238, tv_nsec: 685617646 } })
127382025-04-30T06:42:53.886ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
127392025-04-30T06:42:53.887ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 89 2a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .*..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58789 req_id = 3ebcea10-d9f1-47f7-957b-042f09f0200e
127402025-04-30T06:42:53.887ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35114,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58789 req_id = 3ebcea10-d9f1-47f7-957b-042f09f0200e
127412025-04-30T06:42:53.887ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127422025-04-30T06:42:53.887ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58789 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3ebcea10-d9f1-47f7-957b-042f09f0200e
127432025-04-30T06:42:53.939ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
127442025-04-30T06:42:53.939ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127452025-04-30T06:42:53.940ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127462025-04-30T06:42:53.940ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127472025-04-30T06:42:53.940ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
127482025-04-30T06:42:53.940ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127492025-04-30T06:42:53.940ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
127502025-04-30T06:42:53.941ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
127512025-04-30T06:42:53.942ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
127522025-04-30T06:42:53.942ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
127532025-04-30T06:42:53.943ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e0 3a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .:..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62404 req_id = be0c7789-217a-4c9f-96f7-620c5b065652
127542025-04-30T06:42:53.943ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57402,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62404 req_id = be0c7789-217a-4c9f-96f7-620c5b065652
127552025-04-30T06:42:53.943ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
127562025-04-30T06:42:53.943ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57402, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62404 req_id = be0c7789-217a-4c9f-96f7-620c5b065652
127572025-04-30T06:42:53.943ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
127582025-04-30T06:42:53.944ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
127592025-04-30T06:42:53.944ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
127602025-04-30T06:42:53.944ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127612025-04-30T06:42:53.950ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127622025-04-30T06:42:53.950ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127632025-04-30T06:42:53.950ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
127642025-04-30T06:42:53.950ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127652025-04-30T06:42:53.955ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
127662025-04-30T06:42:53.955ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
127672025-04-30T06:42:53.956ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bb6ea86-d3f4-4bb9-b350-1f3a19a414be", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127682025-04-30T06:42:53.956ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
127692025-04-30T06:42:53.956ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127702025-04-30T06:42:53.956ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
127712025-04-30T06:42:53.957ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127722025-04-30T06:42:53.960ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c7fd92e-8bbc-4cf2-8a6c-d434598439cf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127732025-04-30T06:42:53.960ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
127742025-04-30T06:42:53.961ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
127752025-04-30T06:42:53.961ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
127762025-04-30T06:42:53.964ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b42489b-2c7d-4853-b02c-1c7eae642b09", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127772025-04-30T06:42:53.964ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
127782025-04-30T06:42:53.969ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "578450e8-e753-4bfb-bef2-fa292c9be664", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127792025-04-30T06:42:53.969ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
127802025-04-30T06:42:53.972ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "591c1860-cb8d-4305-86a2-1c24a872ae56", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127812025-04-30T06:42:53.972ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
127822025-04-30T06:42:53.975ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ff7d2c8-b14d-4ff6-9947-c8075c0e8f33", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127832025-04-30T06:42:53.975ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
127842025-04-30T06:42:53.979ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99ac0a43-89b1-4d93-a1ae-488fc06c9dfb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127852025-04-30T06:42:53.980ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
127862025-04-30T06:42:53.982ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d965e729-3579-4255-bd65-1ab2c4b61399", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127872025-04-30T06:42:53.982ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
127882025-04-30T06:42:53.983ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "038e513f-15d8-4349-876a-7f71ba40fb3b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127892025-04-30T06:42:53.983ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
127902025-04-30T06:42:53.986ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c4a92e4-e0ce-44c2-8162-36820f8de0e8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127912025-04-30T06:42:53.986ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
127922025-04-30T06:42:53.988ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e98cc32-fc42-4174-91fe-5b7a1e4bf6e4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127932025-04-30T06:42:53.988ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
127942025-04-30T06:42:53.992ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca693b3f-f9c2-4465-ae05-63a9a321148d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127952025-04-30T06:42:53.992ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
127962025-04-30T06:42:53.993ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a37adb0-d54d-4d44-b3f4-4c3a588c08f7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127972025-04-30T06:42:53.993ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
127982025-04-30T06:42:53.997ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83128e34-623d-43b9-843d-87de70ac0ff1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
127992025-04-30T06:42:53.997ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
128002025-04-30T06:42:54.002ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4acfba69-55d8-4ce2-896a-365eda205a61", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128012025-04-30T06:42:54.003ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
128022025-04-30T06:42:54.008ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7f36a14-6a59-458f-a6ef-603ca02c0626", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128032025-04-30T06:42:54.009ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
128042025-04-30T06:42:54.014ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
128052025-04-30T06:42:54.018ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acc72d88-aa96-41e2-8b39-f1ca26b08e9c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128062025-04-30T06:42:54.018ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
128072025-04-30T06:42:54.021ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4db8e93a-9852-4bcd-872e-68ec1180ca53", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128082025-04-30T06:42:54.021ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
128092025-04-30T06:42:54.028ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e14c9d6-8ac9-4644-a96c-a0c6fc327e7f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128102025-04-30T06:42:54.029ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
128112025-04-30T06:42:54.032ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c37550e4-9748-4fc5-91b4-08b6ab381bef", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128122025-04-30T06:42:54.032ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
128132025-04-30T06:42:54.034ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3574ef83-db41-44ef-a36f-db81057f44ad", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128142025-04-30T06:42:54.037ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
128152025-04-30T06:42:54.037ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94efcfa0-2093-45ae-a70a-223e7efe7d6a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128162025-04-30T06:42:54.039ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
128172025-04-30T06:42:54.041ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e788800-40fd-481a-aa9f-c0e67fa44884", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128182025-04-30T06:42:54.041ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
128192025-04-30T06:42:54.042ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
128202025-04-30T06:42:54.042ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a6f3306-e948-4149-a9de-aba5b4c4084a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128212025-04-30T06:42:54.042ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
128222025-04-30T06:42:54.044ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1eeb4027-71b6-46d7-bacb-bff652ba78b1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128232025-04-30T06:42:54.044ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
128242025-04-30T06:42:54.047ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43fd4b56-eed3-435d-aac5-7c23c677f402", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128252025-04-30T06:42:54.047ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
128262025-04-30T06:42:54.051ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90da00c5-56de-4bda-949d-0e38be348d39", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128272025-04-30T06:42:54.051ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
128282025-04-30T06:42:54.054ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db950bdb-356b-4ef9-a177-bd6ca8b0ba1e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128292025-04-30T06:42:54.054ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
128302025-04-30T06:42:54.058ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fab744f-bf8b-4486-9abb-0e99895eef64", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128312025-04-30T06:42:54.059ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
128322025-04-30T06:42:54.063ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3af8f2c2-52e3-429c-87c3-ae4fd1bbde7c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128332025-04-30T06:42:54.063ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
128342025-04-30T06:42:54.065ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128352025-04-30T06:42:54.065ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128362025-04-30T06:42:54.065ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128372025-04-30T06:42:54.065ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
128382025-04-30T06:42:54.066ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128392025-04-30T06:42:54.073ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128402025-04-30T06:42:54.078ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128412025-04-30T06:42:54.078ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128422025-04-30T06:42:54.078ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
128432025-04-30T06:42:54.078ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128442025-04-30T06:42:54.079ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5e5567f-c650-480e-bc3f-ae5bab02e981", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128452025-04-30T06:42:54.079ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
128462025-04-30T06:42:54.087ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128472025-04-30T06:42:54.087ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
128482025-04-30T06:42:54.087ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128492025-04-30T06:42:54.089ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a2addfe-da82-46bf-8542-927b251953aa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128502025-04-30T06:42:54.089ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
128512025-04-30T06:42:54.098ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e9fc908-fe21-4110-a001-4190534abd8f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128522025-04-30T06:42:54.098ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
128532025-04-30T06:42:54.110ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2df181d1-4e17-4aa7-aab3-04b76f99672d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128542025-04-30T06:42:54.110ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
128552025-04-30T06:42:54.116ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "870ae693-98a6-4a6f-9495-187b1fb4e0b7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128562025-04-30T06:42:54.116ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
128572025-04-30T06:42:54.118ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70ed7b55-cdb9-45c1-b4bc-eb4176352a90", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128582025-04-30T06:42:54.118ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
128592025-04-30T06:42:54.121ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe10ce40-a807-4e5a-a995-d4264fbae18e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128602025-04-30T06:42:54.121ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
128612025-04-30T06:42:54.123ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "385e7128-7d7a-4186-b2be-a63da7912cf2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128622025-04-30T06:42:54.123ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
128632025-04-30T06:42:54.125ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89153c66-f5b3-46dd-a7c4-5c0e5602f584", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128642025-04-30T06:42:54.125ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
128652025-04-30T06:42:54.127ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0010468-0a5e-46c3-9a22-e7eb93d07ac2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128662025-04-30T06:42:54.127ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
128672025-04-30T06:42:54.130ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
128682025-04-30T06:42:54.133ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f798eb2a-67ac-4b8b-bdac-00b8591a2708", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128692025-04-30T06:42:54.133ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
128702025-04-30T06:42:54.138ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f263b52e-e49e-4483-a4f9-38eb499e965e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128712025-04-30T06:42:54.138ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
128722025-04-30T06:42:54.141ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c6ffaba-26aa-47b8-9f20-48a7676b0391", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128732025-04-30T06:42:54.142ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
128742025-04-30T06:42:54.143ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdeb4dde-11d7-4cb1-b678-2f27ff2e11f9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128752025-04-30T06:42:54.144ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
128762025-04-30T06:42:54.145ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b5b56c4-742d-4d35-adb7-70130c1a65d7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
128772025-04-30T06:42:54.146ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128782025-04-30T06:42:54.146ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128792025-04-30T06:42:54.146ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128802025-04-30T06:42:54.146ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128812025-04-30T06:42:54.147ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128822025-04-30T06:42:54.150ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128832025-04-30T06:42:54.154ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128842025-04-30T06:42:54.155ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128852025-04-30T06:42:54.155ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128862025-04-30T06:42:54.155ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128872025-04-30T06:42:54.155ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128882025-04-30T06:42:54.155ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128892025-04-30T06:42:54.158ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128902025-04-30T06:42:54.162ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128912025-04-30T06:42:54.162ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128922025-04-30T06:42:54.162ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128932025-04-30T06:42:54.162ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128942025-04-30T06:42:54.281ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
128952025-04-30T06:42:54.281ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
128962025-04-30T06:42:54.282ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128972025-04-30T06:42:54.282ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128982025-04-30T06:42:54.282ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128992025-04-30T06:42:54.282ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
129002025-04-30T06:42:54.283ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129012025-04-30T06:42:54.283ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:54388 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
129022025-04-30T06:42:54.283ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
129032025-04-30T06:42:54.284ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:54388 req_id = f1dbf83c-e215-4d9b-a407-d0c633651dea sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
129042025-04-30T06:42:54.284ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 675 local_addr = [::1]:62607 method = GET remote_addr = [::1]:54388 req_id = f1dbf83c-e215-4d9b-a407-d0c633651dea response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
129052025-04-30T06:42:54.285ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 82 f2 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:51030 req_id = d611662d-72cc-4035-8abf-c97a5f454402
129062025-04-30T06:42:54.285ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33522,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51030 req_id = d611662d-72cc-4035-8abf-c97a5f454402
129072025-04-30T06:42:54.285ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
129082025-04-30T06:42:54.285ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129092025-04-30T06:42:54.285ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51030 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d611662d-72cc-4035-8abf-c97a5f454402
129102025-04-30T06:42:54.286ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1dbf83c-e215-4d9b-a407-d0c633651dea", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:53 GMT"} })
129112025-04-30T06:42:54.288ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
129122025-04-30T06:42:54.288ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
129132025-04-30T06:42:54.288ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
129142025-04-30T06:42:54.290ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
129152025-04-30T06:42:54.290ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129162025-04-30T06:42:54.290ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129172025-04-30T06:42:54.290ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129182025-04-30T06:42:54.291ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129192025-04-30T06:42:54.291ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129202025-04-30T06:42:54.295ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129212025-04-30T06:42:54.295ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129222025-04-30T06:42:54.295ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
129232025-04-30T06:42:54.296ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129242025-04-30T06:42:54.296ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4239, tv_nsec: 95778515 } })
129252025-04-30T06:42:54.296ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
129262025-04-30T06:42:54.297ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e8 cb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51660 req_id = db6d886a-0763-4a9e-99d7-a02565b138fc
129272025-04-30T06:42:54.297ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59595,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51660 req_id = db6d886a-0763-4a9e-99d7-a02565b138fc
129282025-04-30T06:42:54.297ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129292025-04-30T06:42:54.297ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51660 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = db6d886a-0763-4a9e-99d7-a02565b138fc
129302025-04-30T06:42:54.297ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
129312025-04-30T06:42:54.358ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
129322025-04-30T06:42:54.358ZDEBGtest_delete_snapshot (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
129332025-04-30T06:42:54.359ZDEBGtest_delete_snapshot (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
129342025-04-30T06:42:54.359ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
129352025-04-30T06:42:54.360ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
129362025-04-30T06:42:54.360ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
129372025-04-30T06:42:54.361ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 25 2d 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 %-..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:54444 req_id = b2f13aef-beb0-4de5-ae9c-9c8393ae007a
129382025-04-30T06:42:54.361ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9517,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54444 req_id = b2f13aef-beb0-4de5-ae9c-9c8393ae007a
129392025-04-30T06:42:54.361ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
129402025-04-30T06:42:54.362ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129412025-04-30T06:42:54.362ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54444 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b2f13aef-beb0-4de5-ae9c-9c8393ae007a
129422025-04-30T06:42:54.364ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4239, tv_nsec: 163027424 } })
129432025-04-30T06:42:54.364ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
129442025-04-30T06:42:54.364ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129452025-04-30T06:42:54.370ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
129462025-04-30T06:42:54.371ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129472025-04-30T06:42:54.371ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129482025-04-30T06:42:54.372ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129492025-04-30T06:42:54.372ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129502025-04-30T06:42:54.372ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
129512025-04-30T06:42:54.374ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129522025-04-30T06:42:54.374ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 44 46 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e DF..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61726 req_id = 6eccd79d-ec9b-4d15-a8b9-1240f53dd20d
129532025-04-30T06:42:54.374ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17478,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61726 req_id = 6eccd79d-ec9b-4d15-a8b9-1240f53dd20d
129542025-04-30T06:42:54.374ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129552025-04-30T06:42:54.374ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61726 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6eccd79d-ec9b-4d15-a8b9-1240f53dd20d
129562025-04-30T06:42:54.375ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
129572025-04-30T06:42:54.485ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
129582025-04-30T06:42:54.485ZDEBGtest_delete_snapshot (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
129592025-04-30T06:42:54.485ZDEBGtest_delete_snapshot (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
129602025-04-30T06:42:54.485ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
129612025-04-30T06:42:54.487ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
129622025-04-30T06:42:54.487ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
129632025-04-30T06:42:54.489ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: e5 18 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55025 req_id = b6ced243-eef3-4a1e-8c89-73284bc38c30
129642025-04-30T06:42:54.489ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58648,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55025 req_id = b6ced243-eef3-4a1e-8c89-73284bc38c30
129652025-04-30T06:42:54.489ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
129662025-04-30T06:42:54.489ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129672025-04-30T06:42:54.489ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55025 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b6ced243-eef3-4a1e-8c89-73284bc38c30
129682025-04-30T06:42:54.490ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4239, tv_nsec: 289170847 } })
129692025-04-30T06:42:54.490ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
129702025-04-30T06:42:54.490ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7d 55 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e }U..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56690 req_id = 749f1670-8352-49e8-84fe-c1790e629cdb
129712025-04-30T06:42:54.490ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32085,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56690 req_id = 749f1670-8352-49e8-84fe-c1790e629cdb
129722025-04-30T06:42:54.490ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129732025-04-30T06:42:54.490ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56690 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 749f1670-8352-49e8-84fe-c1790e629cdb
129742025-04-30T06:42:54.491ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
129752025-04-30T06:42:54.578ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
129762025-04-30T06:42:54.578ZDEBGtest_delete_snapshot (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
129772025-04-30T06:42:54.578ZDEBGtest_delete_snapshot (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
129782025-04-30T06:42:54.581ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
129792025-04-30T06:42:54.582ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
129802025-04-30T06:42:54.582ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129812025-04-30T06:42:54.582ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129822025-04-30T06:42:54.587ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129832025-04-30T06:42:54.588ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129842025-04-30T06:42:54.588ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129852025-04-30T06:42:54.588ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129862025-04-30T06:42:54.619ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
129872025-04-30T06:42:54.671ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129882025-04-30T06:42:54.675ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129892025-04-30T06:42:54.676ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129902025-04-30T06:42:54.676ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
129912025-04-30T06:42:54.677ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129922025-04-30T06:42:54.682ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129932025-04-30T06:42:54.691ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129942025-04-30T06:42:54.691ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129952025-04-30T06:42:54.691ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
129962025-04-30T06:42:54.691ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129972025-04-30T06:42:54.693ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
129982025-04-30T06:42:54.693ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
129992025-04-30T06:42:54.693ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 827.497184ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
130002025-04-30T06:42:54.693ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
130012025-04-30T06:42:54.693ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
130022025-04-30T06:42:54.693ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130032025-04-30T06:42:54.693ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130042025-04-30T06:42:54.693ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130052025-04-30T06:42:54.694ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130062025-04-30T06:42:54.700ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130072025-04-30T06:42:54.701ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
130082025-04-30T06:42:54.701ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130092025-04-30T06:42:54.701ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130102025-04-30T06:42:54.708ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
130112025-04-30T06:42:54.708ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130122025-04-30T06:42:54.708ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130132025-04-30T06:42:54.709ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130142025-04-30T06:42:54.713ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
130152025-04-30T06:42:54.713ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
130162025-04-30T06:42:54.715ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6d 96 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 m............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59430 req_id = b7db22f2-3ddf-4ad7-b494-0fc666a2a9cd
130172025-04-30T06:42:54.715ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28054,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59430 req_id = b7db22f2-3ddf-4ad7-b494-0fc666a2a9cd
130182025-04-30T06:42:54.715ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
130192025-04-30T06:42:54.715ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130202025-04-30T06:42:54.715ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59430 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b7db22f2-3ddf-4ad7-b494-0fc666a2a9cd
130212025-04-30T06:42:54.716ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4239, tv_nsec: 514986309 } })
130222025-04-30T06:42:54.717ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
130232025-04-30T06:42:54.717ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 14 f8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55550 req_id = 314ad498-7455-4b8f-b13a-6715f1e6463d
130242025-04-30T06:42:54.718ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5368,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55550 req_id = 314ad498-7455-4b8f-b13a-6715f1e6463d
130252025-04-30T06:42:54.718ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130262025-04-30T06:42:54.718ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55550 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 314ad498-7455-4b8f-b13a-6715f1e6463d
130272025-04-30T06:42:54.830ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
130282025-04-30T06:42:54.831ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b5 b0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51771 req_id = 8865e09a-c1cd-45f3-9946-a0240657bf92
130292025-04-30T06:42:54.832ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46512,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51771 req_id = 8865e09a-c1cd-45f3-9946-a0240657bf92
130302025-04-30T06:42:54.832ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
130312025-04-30T06:42:54.832ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46512, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51771 req_id = 8865e09a-c1cd-45f3-9946-a0240657bf92
130322025-04-30T06:42:54.832ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
130332025-04-30T06:42:54.832ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
130342025-04-30T06:42:54.832ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
130352025-04-30T06:42:54.834ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "811af308-8562-4a9d-bd61-4539f73f66ba", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130362025-04-30T06:42:54.835ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
130372025-04-30T06:42:54.837ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91f16f5d-f58e-4fbf-a768-3e7e2a04fac1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130382025-04-30T06:42:54.837ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
130392025-04-30T06:42:54.840ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "746a05b7-f98c-4a14-90dc-8a7139201833", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130402025-04-30T06:42:54.841ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
130412025-04-30T06:42:54.843ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc25108-e551-42f0-a0ad-d3601b1cfe18", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130422025-04-30T06:42:54.854ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
130432025-04-30T06:42:54.858ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6e6c6b0-cc82-4e41-bf52-7f14ba328a41", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130442025-04-30T06:42:54.858ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
130452025-04-30T06:42:54.865ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
130462025-04-30T06:42:54.867ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ced31ab6-c0fe-4cfe-ba94-5f29ca1192b7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130472025-04-30T06:42:54.867ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
130482025-04-30T06:42:54.884ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e163903-c184-4f30-aa17-378bfc5c596f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130492025-04-30T06:42:54.884ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
130502025-04-30T06:42:54.921ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130512025-04-30T06:42:54.922ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130522025-04-30T06:42:54.922ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130532025-04-30T06:42:54.927ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
130542025-04-30T06:42:54.932ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130552025-04-30T06:42:54.938ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
130562025-04-30T06:42:54.938ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
130572025-04-30T06:42:54.938ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
130582025-04-30T06:42:54.946ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
130592025-04-30T06:42:54.949ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130602025-04-30T06:42:54.971ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130612025-04-30T06:42:54.971ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130622025-04-30T06:42:54.971ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
130632025-04-30T06:42:54.972ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130642025-04-30T06:42:54.977ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
130652025-04-30T06:42:54.977ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
130662025-04-30T06:42:54.981ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f205a557-5d8a-42ea-bc97-f3bcc5167977", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130672025-04-30T06:42:54.981ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
130682025-04-30T06:42:54.983ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
130692025-04-30T06:42:54.983ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
130702025-04-30T06:42:54.984ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130712025-04-30T06:42:54.984ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
130722025-04-30T06:42:54.985ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130732025-04-30T06:42:54.986ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecc29b1f-8a2d-43df-ac07-ae3dced6fb0d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130742025-04-30T06:42:54.986ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
130752025-04-30T06:42:54.988ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8c0a567-b309-4043-b9b7-b01ed7c73401", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130762025-04-30T06:42:54.989ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
130772025-04-30T06:42:54.991ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e598b2e3-791d-44d3-96eb-d0496e0c774e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130782025-04-30T06:42:54.992ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
130792025-04-30T06:42:54.994ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "776778df-8819-49e0-979a-bab87186cf7f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130802025-04-30T06:42:54.994ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
130812025-04-30T06:42:54.996ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41589af6-8ee2-49b5-a773-dde51072d499", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130822025-04-30T06:42:54.996ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
130832025-04-30T06:42:55.007ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ed153f7-2a24-4f7c-9299-0d694aeb1383", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130842025-04-30T06:42:55.007ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
130852025-04-30T06:42:55.013ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dfabe4c-05b5-4a5e-8a58-4866ab1ff502", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130862025-04-30T06:42:55.013ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
130872025-04-30T06:42:55.019ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a45465-aae4-495d-812b-23b43597a1a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130882025-04-30T06:42:55.020ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
130892025-04-30T06:42:55.059ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6995971c-afcc-4bd1-be24-3474faa966ed", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130902025-04-30T06:42:55.059ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
130912025-04-30T06:42:55.060ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
130922025-04-30T06:42:55.073ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4b8ce84-9662-4959-aca9-b0a5209c3fa1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
130932025-04-30T06:42:55.073ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
130942025-04-30T06:42:55.073ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
130952025-04-30T06:42:55.125ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130962025-04-30T06:42:55.125ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130972025-04-30T06:42:55.125ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130982025-04-30T06:42:55.125ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
130992025-04-30T06:42:55.126ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131002025-04-30T06:42:55.153ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131012025-04-30T06:42:55.162ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131022025-04-30T06:42:55.162ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131032025-04-30T06:42:55.162ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
131042025-04-30T06:42:55.163ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131052025-04-30T06:42:55.169ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30add736-de3c-4457-97dd-e819c58b2156", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131062025-04-30T06:42:55.169ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
131072025-04-30T06:42:55.169ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131082025-04-30T06:42:55.169ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
131092025-04-30T06:42:55.170ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131102025-04-30T06:42:55.202ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8b1d15b-2bb8-4081-a082-45a9aca00765", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131112025-04-30T06:42:55.202ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
131122025-04-30T06:42:55.207ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ff3e902-6488-480f-9581-6a91490f4428", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131132025-04-30T06:42:55.208ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
131142025-04-30T06:42:55.210ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "318d658d-78cd-49c1-9d85-5006ae0c428a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131152025-04-30T06:42:55.211ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
131162025-04-30T06:42:55.219ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e66f443-19ed-4b22-9b59-33127a6acb8b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131172025-04-30T06:42:55.219ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
131182025-04-30T06:42:55.220ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20f5dfe5-f762-45d3-bed9-22c17bb4523e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131192025-04-30T06:42:55.220ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
131202025-04-30T06:42:55.225ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9ab66cc-6cc7-43ad-8af4-b72a91b28dfe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131212025-04-30T06:42:55.226ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
131222025-04-30T06:42:55.229ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28c2fdc2-5ebc-48d6-908f-01f0142df53c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131232025-04-30T06:42:55.231ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
131242025-04-30T06:42:55.238ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd16288d-cdbe-4aac-9a53-fcf992c341c2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131252025-04-30T06:42:55.238ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
131262025-04-30T06:42:55.247ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
131272025-04-30T06:42:55.249ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e69f04ac-004f-49a1-884d-f65e543cd8f6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131282025-04-30T06:42:55.249ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
131292025-04-30T06:42:55.254ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaa3f41b-22e0-4c60-95ec-f8fd58ed52e4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131302025-04-30T06:42:55.254ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
131312025-04-30T06:42:55.257ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2af84e6a-38a3-40ae-b38d-f6adadd674bb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131322025-04-30T06:42:55.257ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
131332025-04-30T06:42:55.258ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdae9c62-df5f-442f-95f6-454af8bfbcb0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131342025-04-30T06:42:55.258ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
131352025-04-30T06:42:55.260ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "877e58fd-c4d8-417c-86ee-1151f97641f8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131362025-04-30T06:42:55.260ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
131372025-04-30T06:42:55.261ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c273a130-0368-4621-b30a-42ba0ea3b6fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131382025-04-30T06:42:55.261ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
131392025-04-30T06:42:55.263ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98e096e1-379d-4088-b98c-e0d56a9d9015", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131402025-04-30T06:42:55.263ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
131412025-04-30T06:42:55.267ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b80c0e2-7ada-4090-a4c6-bb9e146fa1ee", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131422025-04-30T06:42:55.268ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
131432025-04-30T06:42:55.272ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdb322d6-5d96-4eda-b553-18841abfd78d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131442025-04-30T06:42:55.272ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
131452025-04-30T06:42:55.275ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71dc8f31-eae2-409e-b9cc-b66e6c7b23fa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131462025-04-30T06:42:55.275ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
131472025-04-30T06:42:55.278ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55002915-ec46-4447-8bca-6282eeb93bef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131482025-04-30T06:42:55.278ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
131492025-04-30T06:42:55.280ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20e06000-1a91-4627-affe-3d85d8c4cb27", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131502025-04-30T06:42:55.280ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
131512025-04-30T06:42:55.283ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5c24a2d-a5cf-4422-8c2e-8e470d16ab96", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131522025-04-30T06:42:55.283ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
131532025-04-30T06:42:55.285ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38d6ee5e-e0d5-476e-b62f-d890394a88bf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131542025-04-30T06:42:55.285ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
131552025-04-30T06:42:55.287ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24279144-635f-4138-8347-fccaa25409b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131562025-04-30T06:42:55.287ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
131572025-04-30T06:42:55.290ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca2ed12c-6730-41b7-908e-340633c5b7f6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131582025-04-30T06:42:55.293ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
131592025-04-30T06:42:55.295ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7f5ac29-02cc-4367-a162-1b6c2459e8a5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131602025-04-30T06:42:55.295ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
131612025-04-30T06:42:55.297ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d463d72-6936-40f8-862b-420d212e0608", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131622025-04-30T06:42:55.297ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
131632025-04-30T06:42:55.297ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131642025-04-30T06:42:55.297ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131652025-04-30T06:42:55.297ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131662025-04-30T06:42:55.298ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131672025-04-30T06:42:55.298ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131682025-04-30T06:42:55.298ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131692025-04-30T06:42:55.298ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131702025-04-30T06:42:55.298ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
131712025-04-30T06:42:55.300ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131722025-04-30T06:42:55.302ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131732025-04-30T06:42:55.305ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
131742025-04-30T06:42:55.305ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
131752025-04-30T06:42:55.305ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131762025-04-30T06:42:55.305ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131772025-04-30T06:42:55.305ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131782025-04-30T06:42:55.306ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131792025-04-30T06:42:55.312ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131802025-04-30T06:42:55.324ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131812025-04-30T06:42:55.325ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131822025-04-30T06:42:55.325ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
131832025-04-30T06:42:55.325ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131842025-04-30T06:42:55.331ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131852025-04-30T06:42:55.331ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
131862025-04-30T06:42:55.331ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131872025-04-30T06:42:55.331ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131882025-04-30T06:42:55.338ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
131892025-04-30T06:42:55.338ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131902025-04-30T06:42:55.338ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131912025-04-30T06:42:55.338ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131922025-04-30T06:42:55.404ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
131932025-04-30T06:42:55.404ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
131942025-04-30T06:42:55.405ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:41204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
131952025-04-30T06:42:55.407ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:41204 req_id = 5f8e7c29-7de2-4829-925a-93b08c21cdaf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
131962025-04-30T06:42:55.407ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 491 local_addr = [::1]:62607 method = GET remote_addr = [::1]:41204 req_id = 5f8e7c29-7de2-4829-925a-93b08c21cdaf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
131972025-04-30T06:42:55.408ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f8e7c29-7de2-4829-925a-93b08c21cdaf", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
131982025-04-30T06:42:55.410ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131992025-04-30T06:42:55.410ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
132002025-04-30T06:42:55.410ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
132012025-04-30T06:42:55.412ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
132022025-04-30T06:42:55.412ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132032025-04-30T06:42:55.412ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132042025-04-30T06:42:55.412ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132052025-04-30T06:42:55.413ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132062025-04-30T06:42:55.429ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132072025-04-30T06:42:55.436ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
132082025-04-30T06:42:55.437ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132092025-04-30T06:42:55.437ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132102025-04-30T06:42:55.437ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132112025-04-30T06:42:55.467ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
132122025-04-30T06:42:55.494ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
132132025-04-30T06:42:55.495ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132142025-04-30T06:42:55.495ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 47c39981-9e36-4b59-94f3-15d7912be06d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132152025-04-30T06:42:55.496ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 47c39981-9e36-4b59-94f3-15d7912be06d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132162025-04-30T06:42:55.496ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 47c39981-9e36-4b59-94f3-15d7912be06d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132172025-04-30T06:42:55.497ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 47c39981-9e36-4b59-94f3-15d7912be06d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132182025-04-30T06:42:55.507ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 11398 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 47c39981-9e36-4b59-94f3-15d7912be06d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
132192025-04-30T06:42:55.509ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "47c39981-9e36-4b59-94f3-15d7912be06d", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
132202025-04-30T06:42:55.509ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
132212025-04-30T06:42:55.510ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2ba1ae80-6948-4123-849d-d6b4f491f78e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
132222025-04-30T06:42:55.511ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2ba1ae80-6948-4123-849d-d6b4f491f78e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
132232025-04-30T06:42:55.511ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2ba1ae80-6948-4123-849d-d6b4f491f78e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
132242025-04-30T06:42:55.512ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2ba1ae80-6948-4123-849d-d6b4f491f78e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
132252025-04-30T06:42:55.519ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 9460 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 2ba1ae80-6948-4123-849d-d6b4f491f78e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
132262025-04-30T06:42:55.519ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ba1ae80-6948-4123-849d-d6b4f491f78e", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:54 GMT"} })
132272025-04-30T06:42:55.519ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132282025-04-30T06:42:55.519ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
132292025-04-30T06:42:55.519ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
132302025-04-30T06:42:55.522ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
132312025-04-30T06:42:55.522ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
132322025-04-30T06:42:55.523ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 828.871252ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
132332025-04-30T06:42:55.523ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
132342025-04-30T06:42:55.523ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
132352025-04-30T06:42:55.523ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132362025-04-30T06:42:55.523ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132372025-04-30T06:42:55.523ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132382025-04-30T06:42:55.524ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132392025-04-30T06:42:55.528ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132402025-04-30T06:42:55.533ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
132412025-04-30T06:42:55.533ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132422025-04-30T06:42:55.533ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132432025-04-30T06:42:55.534ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132442025-04-30T06:42:55.534ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132452025-04-30T06:42:55.534ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132462025-04-30T06:42:55.534ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132472025-04-30T06:42:55.535ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
132482025-04-30T06:42:55.535ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132492025-04-30T06:42:55.537ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
132502025-04-30T06:42:55.537ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
132512025-04-30T06:42:55.538ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9b e4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55851 req_id = 92361dad-03d2-46fb-a43a-62c6f47a077a
132522025-04-30T06:42:55.539ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39908,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55851 req_id = 92361dad-03d2-46fb-a43a-62c6f47a077a
132532025-04-30T06:42:55.539ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
132542025-04-30T06:42:55.539ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132552025-04-30T06:42:55.539ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55851 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 92361dad-03d2-46fb-a43a-62c6f47a077a
132562025-04-30T06:42:55.539ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132572025-04-30T06:42:55.543ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132582025-04-30T06:42:55.543ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132592025-04-30T06:42:55.543ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
132602025-04-30T06:42:55.544ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132612025-04-30T06:42:55.544ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4240, tv_nsec: 343363366 } })
132622025-04-30T06:42:55.544ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
132632025-04-30T06:42:55.545ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 21 45 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e !E..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53422 req_id = 51cb611d-50ab-4f54-8894-c75c370bcbb3
132642025-04-30T06:42:55.545ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8517,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53422 req_id = 51cb611d-50ab-4f54-8894-c75c370bcbb3
132652025-04-30T06:42:55.545ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132662025-04-30T06:42:55.545ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53422 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 51cb611d-50ab-4f54-8894-c75c370bcbb3
132672025-04-30T06:42:55.602ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
132682025-04-30T06:42:55.602ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132692025-04-30T06:42:55.602ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
132702025-04-30T06:42:55.602ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132712025-04-30T06:42:55.603ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132722025-04-30T06:42:55.603ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132732025-04-30T06:42:55.603ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c3 f5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49668 req_id = 7b4e073b-6f34-4537-b264-0cc9e55c1c5c
132742025-04-30T06:42:55.603ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50165,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49668 req_id = 7b4e073b-6f34-4537-b264-0cc9e55c1c5c
132752025-04-30T06:42:55.604ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
132762025-04-30T06:42:55.604ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50165, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49668 req_id = 7b4e073b-6f34-4537-b264-0cc9e55c1c5c
132772025-04-30T06:42:55.629ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
132782025-04-30T06:42:55.629ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
132792025-04-30T06:42:55.629ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
132802025-04-30T06:42:55.630ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132812025-04-30T06:42:55.631ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132822025-04-30T06:42:55.631ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132832025-04-30T06:42:55.631ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132842025-04-30T06:42:55.643ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8de87405-69c3-4abe-b163-407fe64d77b1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
132852025-04-30T06:42:55.643ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
132862025-04-30T06:42:55.646ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ab95697-58d3-479b-a8aa-0eae84a0906e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
132872025-04-30T06:42:55.648ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
132882025-04-30T06:42:55.666ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6baad03-098f-483f-a242-179da5c06c1d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
132892025-04-30T06:42:55.666ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
132902025-04-30T06:42:55.673ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bac4553-b8df-464d-91ad-3dc40390d368", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
132912025-04-30T06:42:55.673ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
132922025-04-30T06:42:55.680ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f72bd48-8957-4456-93a0-2def6cc98bf0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
132932025-04-30T06:42:55.680ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
132942025-04-30T06:42:55.684ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
132952025-04-30T06:42:55.687ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fffa9998-f3db-4476-893a-e44dcd53f4b4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
132962025-04-30T06:42:55.687ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
132972025-04-30T06:42:55.691ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10fdab45-78b3-40ba-85df-0d0dfd357ddb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
132982025-04-30T06:42:55.691ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
132992025-04-30T06:42:55.694ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e755a435-8379-4267-b565-c5fbf8b2e25a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133002025-04-30T06:42:55.695ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
133012025-04-30T06:42:55.696ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8ab7a62-f079-447b-bee3-0fb491d98749", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133022025-04-30T06:42:55.696ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
133032025-04-30T06:42:55.698ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "285c3176-54c7-4d8d-82e9-7404cbf7fbbc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133042025-04-30T06:42:55.698ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
133052025-04-30T06:42:55.700ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a48c859-8f72-4eb8-8af8-ae78fc56c094", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133062025-04-30T06:42:55.700ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
133072025-04-30T06:42:55.701ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "000c6e1c-cfb8-47b0-8d97-f3f81ac53a64", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133082025-04-30T06:42:55.701ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
133092025-04-30T06:42:55.704ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "923c0150-6732-4c94-8cfe-cd768718c34a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133102025-04-30T06:42:55.705ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
133112025-04-30T06:42:55.709ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7344a5aa-e082-429c-8d3e-aabd2212aa5c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133122025-04-30T06:42:55.709ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
133132025-04-30T06:42:55.712ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c0e2ab7-0167-483b-ad44-21c864b7060f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133142025-04-30T06:42:55.712ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
133152025-04-30T06:42:55.716ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61f20b68-81f1-4692-8b2b-936b2550e0f1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133162025-04-30T06:42:55.716ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
133172025-04-30T06:42:55.725ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c5fb0fb-c6a2-4fa6-8d80-46cfda43ff93", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133182025-04-30T06:42:55.726ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
133192025-04-30T06:42:55.732ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09c4c7bb-b45e-4909-ac57-2eddaf44e8c4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133202025-04-30T06:42:55.732ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
133212025-04-30T06:42:55.735ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133222025-04-30T06:42:55.735ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133232025-04-30T06:42:55.735ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133242025-04-30T06:42:55.735ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
133252025-04-30T06:42:55.736ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133262025-04-30T06:42:55.739ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b747b51-acf2-44db-ad7e-d7a2950e7943", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133272025-04-30T06:42:55.739ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
133282025-04-30T06:42:55.740ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133292025-04-30T06:42:55.745ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133302025-04-30T06:42:55.745ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133312025-04-30T06:42:55.745ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
133322025-04-30T06:42:55.745ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133332025-04-30T06:42:55.750ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f85d64c7-e593-4f8b-a5af-2aa45a6df4fb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133342025-04-30T06:42:55.753ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
133352025-04-30T06:42:55.761ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133362025-04-30T06:42:55.761ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
133372025-04-30T06:42:55.762ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133382025-04-30T06:42:55.763ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78d9a933-4394-4492-810f-a8c448a625bb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133392025-04-30T06:42:55.764ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
133402025-04-30T06:42:55.765ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea7f0e9b-dc87-48d2-aa4a-eedf1c7d3611", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133412025-04-30T06:42:55.765ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
133422025-04-30T06:42:55.767ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c7197a9-c101-4269-a3f9-c10e1f59b0bb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133432025-04-30T06:42:55.767ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
133442025-04-30T06:42:55.770ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cd1180a-8f9f-49f3-bb1d-ba9d85a2748e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133452025-04-30T06:42:55.770ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
133462025-04-30T06:42:55.772ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f0ac634-3ca6-42b8-b240-1887d6c6bee2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133472025-04-30T06:42:55.773ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
133482025-04-30T06:42:55.777ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef23ae53-71df-4c51-b2fe-a76218b8a310", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133492025-04-30T06:42:55.777ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
133502025-04-30T06:42:55.782ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5d7a540-2c5c-4e79-903a-dbf8c957d48d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133512025-04-30T06:42:55.782ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
133522025-04-30T06:42:55.785ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e499ec1-d7e1-4772-b7ce-331392eef65c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133532025-04-30T06:42:55.785ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
133542025-04-30T06:42:55.791ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff0e864e-9d7c-43f4-b171-4aa15a2429b6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133552025-04-30T06:42:55.791ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
133562025-04-30T06:42:55.792ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
133572025-04-30T06:42:55.795ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b442ed3-c79c-433a-845f-49923919a8be", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133582025-04-30T06:42:55.795ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
133592025-04-30T06:42:55.796ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bb21bdf-a014-4e33-836f-31f28c4b5c66", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133602025-04-30T06:42:55.796ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
133612025-04-30T06:42:55.798ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65887793-9c43-4e06-849d-e42629937945", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133622025-04-30T06:42:55.798ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
133632025-04-30T06:42:55.801ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "799112ad-20cb-406a-b5f7-bff173f1f924", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133642025-04-30T06:42:55.801ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
133652025-04-30T06:42:55.803ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceadfdca-087c-485b-8d36-9cd2ea2054e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133662025-04-30T06:42:55.803ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
133672025-04-30T06:42:55.805ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0efc9573-fd3d-4910-9b9b-d25eda901903", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133682025-04-30T06:42:55.805ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
133692025-04-30T06:42:55.807ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74ddbcb3-a0f1-443a-8a6a-a1ee9c02939f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133702025-04-30T06:42:55.807ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
133712025-04-30T06:42:55.810ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d36aba2-a79a-45d2-9c1e-03312b5b72fc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133722025-04-30T06:42:55.810ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
133732025-04-30T06:42:55.812ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4199c6b-da27-45b3-b7f1-4468d18c3c3a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133742025-04-30T06:42:55.812ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
133752025-04-30T06:42:55.815ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4227a184-0763-4702-9e2a-b3a032a403b9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133762025-04-30T06:42:55.815ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
133772025-04-30T06:42:55.820ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90d65039-b7eb-4bc6-ab5c-d128289fa495", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133782025-04-30T06:42:55.820ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
133792025-04-30T06:42:55.823ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5137a48c-26bd-4d4f-aaf0-3a411e321a18", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133802025-04-30T06:42:55.823ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
133812025-04-30T06:42:55.825ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f7ba4f-577f-4717-8eaa-8c1c8def4eff", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133822025-04-30T06:42:55.825ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
133832025-04-30T06:42:55.827ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51ef110c-6a90-4763-9f73-3745bacdaad2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133842025-04-30T06:42:55.827ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
133852025-04-30T06:42:55.828ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ed41b20-d5e9-4e09-ab20-f0d40f28d36c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133862025-04-30T06:42:55.828ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
133872025-04-30T06:42:55.830ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6362315b-f859-4770-abad-fd28f422a4e9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
133882025-04-30T06:42:55.830ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
133892025-04-30T06:42:55.830ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133902025-04-30T06:42:55.830ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133912025-04-30T06:42:55.830ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133922025-04-30T06:42:55.831ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133932025-04-30T06:42:55.849ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133942025-04-30T06:42:55.849ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133952025-04-30T06:42:55.849ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133962025-04-30T06:42:55.849ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
133972025-04-30T06:42:55.850ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133982025-04-30T06:42:55.857ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133992025-04-30T06:42:55.861ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134002025-04-30T06:42:55.861ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134012025-04-30T06:42:55.861ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134022025-04-30T06:42:55.861ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134032025-04-30T06:42:55.861ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134042025-04-30T06:42:55.862ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134052025-04-30T06:42:55.864ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134062025-04-30T06:42:55.871ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134072025-04-30T06:42:55.871ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134082025-04-30T06:42:55.871ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
134092025-04-30T06:42:55.871ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134102025-04-30T06:42:55.874ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134112025-04-30T06:42:55.877ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134122025-04-30T06:42:55.877ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134132025-04-30T06:42:55.877ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134142025-04-30T06:42:55.878ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134152025-04-30T06:42:55.881ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134162025-04-30T06:42:55.881ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
134172025-04-30T06:42:55.882ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134182025-04-30T06:42:55.960ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
134192025-04-30T06:42:55.960ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
134202025-04-30T06:42:55.961ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:64239 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
134212025-04-30T06:42:55.961ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
134222025-04-30T06:42:55.961ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
134232025-04-30T06:42:55.962ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
134242025-04-30T06:42:55.963ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
134252025-04-30T06:42:55.963ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:64239 req_id = 90749947-4fc9-4f98-8091-40e013a6b47b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
134262025-04-30T06:42:55.964ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 566 local_addr = [::1]:62607 method = GET remote_addr = [::1]:64239 req_id = 90749947-4fc9-4f98-8091-40e013a6b47b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
134272025-04-30T06:42:55.964ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "90749947-4fc9-4f98-8091-40e013a6b47b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
134282025-04-30T06:42:55.966ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
134292025-04-30T06:42:55.966ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
134302025-04-30T06:42:55.966ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
134312025-04-30T06:42:55.968ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
134322025-04-30T06:42:55.968ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134332025-04-30T06:42:55.968ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134342025-04-30T06:42:55.968ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134352025-04-30T06:42:55.970ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134362025-04-30T06:42:55.974ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134372025-04-30T06:42:55.980ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
134382025-04-30T06:42:55.980ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134392025-04-30T06:42:55.980ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134402025-04-30T06:42:55.981ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134412025-04-30T06:42:55.982ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
134422025-04-30T06:42:55.982ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
134432025-04-30T06:42:55.988ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
134442025-04-30T06:42:55.988ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
134452025-04-30T06:42:56.015ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
134462025-04-30T06:42:56.020ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
134472025-04-30T06:42:56.020ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
134482025-04-30T06:42:56.021ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 497.972066ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
134492025-04-30T06:42:56.021ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
134502025-04-30T06:42:56.021ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
134512025-04-30T06:42:56.021ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134522025-04-30T06:42:56.021ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134532025-04-30T06:42:56.021ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134542025-04-30T06:42:56.024ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134552025-04-30T06:42:56.041ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134562025-04-30T06:42:56.049ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
134572025-04-30T06:42:56.049ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134582025-04-30T06:42:56.049ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134592025-04-30T06:42:56.050ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134602025-04-30T06:42:56.056ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
134612025-04-30T06:42:56.056ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
134622025-04-30T06:42:56.057ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 93 64 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .d..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62324 req_id = 0dd81956-988b-4b4b-91cb-67645e6db59a
134632025-04-30T06:42:56.057ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37732,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62324 req_id = 0dd81956-988b-4b4b-91cb-67645e6db59a
134642025-04-30T06:42:56.057ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
134652025-04-30T06:42:56.057ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134662025-04-30T06:42:56.057ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62324 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0dd81956-988b-4b4b-91cb-67645e6db59a
134672025-04-30T06:42:56.058ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4240, tv_nsec: 856924754 } })
134682025-04-30T06:42:56.058ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
134692025-04-30T06:42:56.058ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 49 bb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e I...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64556 req_id = 12a3d53a-f57d-427e-9b98-ac4a42bcaf62
134702025-04-30T06:42:56.058ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18875,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64556 req_id = 12a3d53a-f57d-427e-9b98-ac4a42bcaf62
134712025-04-30T06:42:56.058ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134722025-04-30T06:42:56.058ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64556 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 12a3d53a-f57d-427e-9b98-ac4a42bcaf62
134732025-04-30T06:42:56.106ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
134742025-04-30T06:42:56.106ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134752025-04-30T06:42:56.106ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134762025-04-30T06:42:56.106ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134772025-04-30T06:42:56.107ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
134782025-04-30T06:42:56.107ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134792025-04-30T06:42:56.109ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
134802025-04-30T06:42:56.110ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 62 8d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c b............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59233 req_id = c450a687-d3db-418c-8690-fdd85e2821b1
134812025-04-30T06:42:56.110ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25229,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59233 req_id = c450a687-d3db-418c-8690-fdd85e2821b1
134822025-04-30T06:42:56.110ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
134832025-04-30T06:42:56.110ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25229, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59233 req_id = c450a687-d3db-418c-8690-fdd85e2821b1
134842025-04-30T06:42:56.112ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
134852025-04-30T06:42:56.113ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
134862025-04-30T06:42:56.113ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
134872025-04-30T06:42:56.113ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134882025-04-30T06:42:56.119ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134892025-04-30T06:42:56.119ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134902025-04-30T06:42:56.119ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
134912025-04-30T06:42:56.119ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134922025-04-30T06:42:56.125ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134932025-04-30T06:42:56.125ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
134942025-04-30T06:42:56.125ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134952025-04-30T06:42:56.136ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e5d9d2c-dc7d-4fd7-9b11-338739ae0cbd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
134962025-04-30T06:42:56.137ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
134972025-04-30T06:42:56.149ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "631932a6-eb47-4a86-ba37-a849cc4c13a0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
134982025-04-30T06:42:56.149ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
134992025-04-30T06:42:56.158ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6c789e0-e659-480f-8771-d87e40213f9e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135002025-04-30T06:42:56.158ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
135012025-04-30T06:42:56.161ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e5996a8-5c3a-470c-822a-b1fe46ae6f4e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135022025-04-30T06:42:56.161ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
135032025-04-30T06:42:56.165ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a9fc53a-74e5-4793-a3e4-4bb180513457", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135042025-04-30T06:42:56.165ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
135052025-04-30T06:42:56.169ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf38d590-1fda-4b44-8b70-3c4fc8a68ec4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135062025-04-30T06:42:56.169ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
135072025-04-30T06:42:56.176ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
135082025-04-30T06:42:56.177ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1da567c-f341-46c9-be9c-a64a001a1759", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135092025-04-30T06:42:56.177ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
135102025-04-30T06:42:56.181ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7450e733-fef8-462c-a118-21c3f37d3ebe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135112025-04-30T06:42:56.181ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
135122025-04-30T06:42:56.184ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4062f004-82d5-4b9b-aef8-a196607eff4a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135132025-04-30T06:42:56.184ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
135142025-04-30T06:42:56.187ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9eb07f0-f730-4547-8c8f-f1d71fb4856b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135152025-04-30T06:42:56.188ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
135162025-04-30T06:42:56.189ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24c60b9d-f058-4376-baf3-99df8060ad20", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135172025-04-30T06:42:56.189ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
135182025-04-30T06:42:56.190ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "371f41e5-1413-4936-93ee-663b472aef6d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135192025-04-30T06:42:56.190ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
135202025-04-30T06:42:56.192ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2837039d-b46e-4b6b-9e4b-8ee06814ab45", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135212025-04-30T06:42:56.192ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
135222025-04-30T06:42:56.194ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:39244 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
135232025-04-30T06:42:56.194ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:39244 req_id = 280ffac2-5df6-46bf-8c78-085c8a1c9a31 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
135242025-04-30T06:42:56.196ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 1886 local_addr = [::1]:33598 method = GET remote_addr = [::1]:39244 req_id = 280ffac2-5df6-46bf-8c78-085c8a1c9a31 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
135252025-04-30T06:42:56.198ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5adb823a-02ac-4339-b890-231e1a72f4fc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135262025-04-30T06:42:56.198ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
135272025-04-30T06:42:56.201ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16a357bd-5a02-4ff2-9a74-62a084d67f58", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135282025-04-30T06:42:56.201ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
135292025-04-30T06:42:56.204ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "daae0a34-ab1d-408f-8908-b8ea58f02afc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135302025-04-30T06:42:56.204ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
135312025-04-30T06:42:56.207ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26e7177f-dda8-4c07-bdf1-a7ca193bf01b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135322025-04-30T06:42:56.207ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
135332025-04-30T06:42:56.209ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0683f05f-fd53-4ea9-91c1-a2942c22c796", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135342025-04-30T06:42:56.209ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
135352025-04-30T06:42:56.213ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d940455-fbef-48bb-a10a-b0a805753f1a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135362025-04-30T06:42:56.213ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
135372025-04-30T06:42:56.214ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7de26301-d262-4049-a0cf-69cf12545c5d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135382025-04-30T06:42:56.214ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
135392025-04-30T06:42:56.216ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "469d2bf4-987b-440c-9ec7-bb3cc9525ef9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135402025-04-30T06:42:56.216ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
135412025-04-30T06:42:56.217ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "317eaa9a-bc71-45d7-8aa5-f7159a7927b2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135422025-04-30T06:42:56.217ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
135432025-04-30T06:42:56.218ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af6d2e8-20b1-436d-8c9b-a634928b1fc5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135442025-04-30T06:42:56.218ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
135452025-04-30T06:42:56.220ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d834df5b-15d5-42ee-b7af-12a11a8c0e53", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135462025-04-30T06:42:56.220ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
135472025-04-30T06:42:56.222ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9a52b22-0241-4d91-b7a5-f63dfeca7b1a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135482025-04-30T06:42:56.222ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
135492025-04-30T06:42:56.224ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a91014a5-959c-4b52-a5a7-11a9d0c28095", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135502025-04-30T06:42:56.224ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
135512025-04-30T06:42:56.227ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135522025-04-30T06:42:56.227ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135532025-04-30T06:42:56.227ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135542025-04-30T06:42:56.227ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
135552025-04-30T06:42:56.229ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135562025-04-30T06:42:56.230ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a9dc520-5a56-4ad8-9b89-29c15f65ec8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135572025-04-30T06:42:56.230ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
135582025-04-30T06:42:56.232ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135592025-04-30T06:42:56.244ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135602025-04-30T06:42:56.244ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135612025-04-30T06:42:56.244ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
135622025-04-30T06:42:56.245ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135632025-04-30T06:42:56.245ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3fd8a60-ba42-4cf6-a857-1350fd1e2488", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135642025-04-30T06:42:56.245ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
135652025-04-30T06:42:56.250ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135662025-04-30T06:42:56.250ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
135672025-04-30T06:42:56.250ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135682025-04-30T06:42:56.251ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e178ddac-a47c-4579-86fb-349807d75183", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135692025-04-30T06:42:56.251ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
135702025-04-30T06:42:56.257ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "998b452c-9dc9-45ac-b811-912d2f875c55", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135712025-04-30T06:42:56.257ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
135722025-04-30T06:42:56.259ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f42decbd-d3d7-4e67-92c5-2c3225bbb922", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135732025-04-30T06:42:56.260ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
135742025-04-30T06:42:56.262ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "470eee88-0ad4-4182-85a0-23c322b035ce", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135752025-04-30T06:42:56.262ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
135762025-04-30T06:42:56.269ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d215de81-32c4-4580-9c98-a5ffe72dfd7c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135772025-04-30T06:42:56.270ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
135782025-04-30T06:42:56.274ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e474ba4-a19d-4c63-b01f-9792abb2f25b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135792025-04-30T06:42:56.274ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
135802025-04-30T06:42:56.281ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4720c7fa-8700-412b-939b-26517c37231a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135812025-04-30T06:42:56.281ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
135822025-04-30T06:42:56.300ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "928ec064-270d-46e7-89a7-d9d95cbe4ce3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135832025-04-30T06:42:56.300ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
135842025-04-30T06:42:56.301ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
135852025-04-30T06:42:56.304ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3697986-d1fc-46a6-b58a-2e40fd395df6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135862025-04-30T06:42:56.305ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
135872025-04-30T06:42:56.307ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c56e8388-157f-4fec-bac8-819b078d1a73", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135882025-04-30T06:42:56.308ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
135892025-04-30T06:42:56.310ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceb495a9-c157-470e-bdf8-39899c366ebf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135902025-04-30T06:42:56.311ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
135912025-04-30T06:42:56.313ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff98b39-714e-4f9c-aaf8-956a04bab4c4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135922025-04-30T06:42:56.313ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
135932025-04-30T06:42:56.314ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75c4284e-f191-480f-a085-69903cce0054", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135942025-04-30T06:42:56.315ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
135952025-04-30T06:42:56.316ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "099539e0-d67a-4848-a897-21a5dec1737d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135962025-04-30T06:42:56.316ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
135972025-04-30T06:42:56.320ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae011f29-4328-43f1-a306-98a0a8fc4ba0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
135982025-04-30T06:42:56.320ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
135992025-04-30T06:42:56.337ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "235dae68-8946-494e-a9c8-4246740bd444", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
136002025-04-30T06:42:56.337ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
136012025-04-30T06:42:56.340ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2543dd63-0aae-49e9-b61a-35af1eb5f820", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
136022025-04-30T06:42:56.340ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136032025-04-30T06:42:56.340ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136042025-04-30T06:42:56.340ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136052025-04-30T06:42:56.340ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136062025-04-30T06:42:56.342ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136072025-04-30T06:42:56.347ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136082025-04-30T06:42:56.350ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136092025-04-30T06:42:56.350ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136102025-04-30T06:42:56.350ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136112025-04-30T06:42:56.350ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136122025-04-30T06:42:56.350ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136132025-04-30T06:42:56.351ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136142025-04-30T06:42:56.352ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136152025-04-30T06:42:56.352ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136162025-04-30T06:42:56.352ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136172025-04-30T06:42:56.352ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
136182025-04-30T06:42:56.353ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136192025-04-30T06:42:56.356ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136202025-04-30T06:42:56.360ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136212025-04-30T06:42:56.360ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136222025-04-30T06:42:56.360ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136232025-04-30T06:42:56.360ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136242025-04-30T06:42:56.361ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
136252025-04-30T06:42:56.361ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c5 38 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .8..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:65302 req_id = cfdbf64c-3e8b-45bf-a8ee-87e97c2da06b
136262025-04-30T06:42:56.361ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50488,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65302 req_id = cfdbf64c-3e8b-45bf-a8ee-87e97c2da06b
136272025-04-30T06:42:56.362ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
136282025-04-30T06:42:56.362ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136292025-04-30T06:42:56.362ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65302 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cfdbf64c-3e8b-45bf-a8ee-87e97c2da06b
136302025-04-30T06:42:56.362ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136312025-04-30T06:42:56.368ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136322025-04-30T06:42:56.368ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136332025-04-30T06:42:56.368ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
136342025-04-30T06:42:56.369ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136352025-04-30T06:42:56.369ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 168384747 } })
136362025-04-30T06:42:56.369ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
136372025-04-30T06:42:56.441ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
136382025-04-30T06:42:56.441ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
136392025-04-30T06:42:56.443ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:54513 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
136402025-04-30T06:42:56.443ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:54513 req_id = 5bce31a0-9061-4c9d-b97e-16ebc34b7410 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136412025-04-30T06:42:56.444ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7c d4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e |...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62250 req_id = 8bb7c3ad-1eb4-430f-8171-b5c2d072605b
136422025-04-30T06:42:56.444ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31956,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62250 req_id = 8bb7c3ad-1eb4-430f-8171-b5c2d072605b
136432025-04-30T06:42:56.444ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136442025-04-30T06:42:56.444ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62250 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8bb7c3ad-1eb4-430f-8171-b5c2d072605b
136452025-04-30T06:42:56.445ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 1035 local_addr = [::1]:62607 method = GET remote_addr = [::1]:54513 req_id = 5bce31a0-9061-4c9d-b97e-16ebc34b7410 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136462025-04-30T06:42:56.445ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136472025-04-30T06:42:56.446ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
136482025-04-30T06:42:56.446ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136492025-04-30T06:42:56.446ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
136502025-04-30T06:42:56.511ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
136512025-04-30T06:42:56.511ZDEBGtest_delete_snapshot (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
136522025-04-30T06:42:56.511ZDEBGtest_delete_snapshot (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
136532025-04-30T06:42:56.511ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
136542025-04-30T06:42:56.513ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
136552025-04-30T06:42:56.513ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bce31a0-9061-4c9d-b97e-16ebc34b7410", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
136562025-04-30T06:42:56.519ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136572025-04-30T06:42:56.519ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136582025-04-30T06:42:56.520ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
136592025-04-30T06:42:56.527ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
136602025-04-30T06:42:56.527ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136612025-04-30T06:42:56.527ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136622025-04-30T06:42:56.527ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136632025-04-30T06:42:56.527ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136642025-04-30T06:42:56.528ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
136652025-04-30T06:42:56.530ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ec 38 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .8..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:51084 req_id = 557efb00-3a37-46f0-97c3-afaeba715e9a
136662025-04-30T06:42:56.530ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60472,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51084 req_id = 557efb00-3a37-46f0-97c3-afaeba715e9a
136672025-04-30T06:42:56.530ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
136682025-04-30T06:42:56.530ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136692025-04-30T06:42:56.530ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51084 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 557efb00-3a37-46f0-97c3-afaeba715e9a
136702025-04-30T06:42:56.530ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 329194741 } })
136712025-04-30T06:42:56.530ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
136722025-04-30T06:42:56.530ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1d 98 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54123 req_id = aeba68c0-6115-44fa-a503-590ef091ce24
136732025-04-30T06:42:56.530ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7576,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54123 req_id = aeba68c0-6115-44fa-a503-590ef091ce24
136742025-04-30T06:42:56.531ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136752025-04-30T06:42:56.531ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54123 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = aeba68c0-6115-44fa-a503-590ef091ce24
136762025-04-30T06:42:56.531ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
136772025-04-30T06:42:56.582ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
136782025-04-30T06:42:56.582ZDEBGtest_delete_snapshot (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
136792025-04-30T06:42:56.582ZDEBGtest_delete_snapshot (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
136802025-04-30T06:42:56.583ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
136812025-04-30T06:42:56.583ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
136822025-04-30T06:42:56.584ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136832025-04-30T06:42:56.589ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
136842025-04-30T06:42:56.589ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136852025-04-30T06:42:56.589ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136862025-04-30T06:42:56.590ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136872025-04-30T06:42:56.591ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136882025-04-30T06:42:56.591ZTRACtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
136892025-04-30T06:42:56.591ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136902025-04-30T06:42:56.594ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 08 d6 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:50629 req_id = ab24d749-31df-4755-80f6-e79d72775f06
136912025-04-30T06:42:56.595ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2262,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50629 req_id = ab24d749-31df-4755-80f6-e79d72775f06
136922025-04-30T06:42:56.595ZDEBGtest_delete_snapshot (store): query key key = _dendrite._tcp
136932025-04-30T06:42:56.597ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136942025-04-30T06:42:56.597ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50629 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ab24d749-31df-4755-80f6-e79d72775f06
136952025-04-30T06:42:56.597ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136962025-04-30T06:42:56.598ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136972025-04-30T06:42:56.598ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136982025-04-30T06:42:56.598ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136992025-04-30T06:42:56.600ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 65050, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 398986006 } })
137002025-04-30T06:42:56.600ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
137012025-04-30T06:42:56.600ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 78 ca 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e x...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61074 req_id = 82f34bea-cda6-4d53-b127-bfd0340c2787
137022025-04-30T06:42:56.600ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30922,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61074 req_id = 82f34bea-cda6-4d53-b127-bfd0340c2787
137032025-04-30T06:42:56.601ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137042025-04-30T06:42:56.601ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61074 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 82f34bea-cda6-4d53-b127-bfd0340c2787
137052025-04-30T06:42:56.601ZINFOtest_delete_snapshot (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
137062025-04-30T06:42:56.654ZINFOtest_delete_snapshot (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
137072025-04-30T06:42:56.654ZDEBGtest_delete_snapshot (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
137082025-04-30T06:42:56.680ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
137092025-04-30T06:42:56.682ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
137102025-04-30T06:42:56.683ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
137112025-04-30T06:42:56.683ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 662.271603ms iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
137122025-04-30T06:42:56.683ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
137132025-04-30T06:42:56.683ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
137142025-04-30T06:42:56.683ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137152025-04-30T06:42:56.683ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137162025-04-30T06:42:56.683ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137172025-04-30T06:42:56.686ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137182025-04-30T06:42:56.690ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137192025-04-30T06:42:56.696ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
137202025-04-30T06:42:56.696ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137212025-04-30T06:42:56.696ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137222025-04-30T06:42:56.696ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137232025-04-30T06:42:56.702ZDEBGtest_delete_snapshot (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(6e5eb653-feb7-4a7f-81f2-73b2ee16696b, 0)]
137242025-04-30T06:42:56.702ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
137252025-04-30T06:42:56.704ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e1 35 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .5..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55029 req_id = 6e2fb67c-8fd9-4053-a6d6-ee81960cb7d4
137262025-04-30T06:42:56.705ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57653,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55029 req_id = 6e2fb67c-8fd9-4053-a6d6-ee81960cb7d4
137272025-04-30T06:42:56.705ZDEBGtest_delete_snapshot (store): query key key = _mgs._tcp
137282025-04-30T06:42:56.705ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137292025-04-30T06:42:56.705ZDEBGtest_delete_snapshot (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55029 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6e2fb67c-8fd9-4053-a6d6-ee81960cb7d4
137302025-04-30T06:42:56.706ZDEBGtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35772, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4241, tv_nsec: 505253797 } })
137312025-04-30T06:42:56.706ZTRACtest_delete_snapshot (DnsResolver): lookup_service_targets: looking up SRV target
137322025-04-30T06:42:56.708ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b0 37 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .7..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50595 req_id = a6b2f3e2-6f82-4db1-8c4e-01d6946f0c69
137332025-04-30T06:42:56.709ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45111,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50595 req_id = a6b2f3e2-6f82-4db1-8c4e-01d6946f0c69
137342025-04-30T06:42:56.709ZDEBGtest_delete_snapshot (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137352025-04-30T06:42:56.709ZDEBGtest_delete_snapshot (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50595 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a6b2f3e2-6f82-4db1-8c4e-01d6946f0c69
137362025-04-30T06:42:56.765ZTRACtest_delete_snapshot (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
137372025-04-30T06:42:56.766ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137382025-04-30T06:42:56.766ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137392025-04-30T06:42:56.766ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137402025-04-30T06:42:56.766ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
137412025-04-30T06:42:56.851ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137422025-04-30T06:42:56.854ZTRACtest_delete_snapshot (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e3 0e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59745 req_id = cbbe95d5-207a-4cf7-a051-7eef6ddffc94
137432025-04-30T06:42:56.854ZDEBGtest_delete_snapshot (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58126,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59745 req_id = cbbe95d5-207a-4cf7-a051-7eef6ddffc94
137442025-04-30T06:42:56.854ZDEBGtest_delete_snapshot (store): query key key = _clickhouse-admin-keeper._tcp
137452025-04-30T06:42:56.854ZERROtest_delete_snapshot (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58126, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59745 req_id = cbbe95d5-207a-4cf7-a051-7eef6ddffc94
137462025-04-30T06:42:57.283ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:61447 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137472025-04-30T06:42:57.283ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:45056 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137482025-04-30T06:42:57.283ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:58190 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137492025-04-30T06:42:57.283ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:48197 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137502025-04-30T06:42:57.283ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:64323 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137512025-04-30T06:42:57.283ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:56371 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137522025-04-30T06:42:57.283ZINFOtest_delete_snapshot (dropshot): accepted connection kind = pantry local_addr = [::1]:33598 remote_addr = [::1]:43274 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137532025-04-30T06:42:57.284ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection background_task = inventory_collection
137542025-04-30T06:42:57.284ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:35772
137552025-04-30T06:42:57.284ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/ignition
137562025-04-30T06:42:57.284ZDEBGtest_delete_snapshot (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137572025-04-30T06:42:57.284ZDEBGtest_delete_snapshot (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137582025-04-30T06:42:57.285ZDEBGtest_delete_snapshot (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69
137592025-04-30T06:42:57.286ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
137602025-04-30T06:42:57.287ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137612025-04-30T06:42:57.292ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137622025-04-30T06:42:57.292ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137632025-04-30T06:42:57.292ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
137642025-04-30T06:42:57.293ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137652025-04-30T06:42:57.293ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:61447 req_id = 83ee142a-b189-4a64-9c7c-e3f693846722 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137662025-04-30T06:42:57.294ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:45056 req_id = b5f08967-a77d-4503-b9aa-54a3c0735978 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137672025-04-30T06:42:57.294ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:58190 req_id = c0dc95c9-47c3-41d9-9ed2-a4a4d6826eb5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137682025-04-30T06:42:57.294ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:48197 req_id = 006ac379-c225-422b-b234-4b78832098e9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137692025-04-30T06:42:57.294ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:64323 req_id = 6fed0ce1-a9a2-43eb-b7b6-955134411a16 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137702025-04-30T06:42:57.294ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:56371 req_id = d51da3b4-602d-47f6-a3fd-3630ec69ae36 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137712025-04-30T06:42:57.295ZTRACtest_delete_snapshot (dropshot): incoming request kind = pantry local_addr = [::1]:33598 method = GET remote_addr = [::1]:43274 req_id = d79341df-2884-43cd-8261-4985ceae3f10 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137722025-04-30T06:42:57.296ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 2585 local_addr = [::1]:33598 method = GET remote_addr = [::1]:61447 req_id = 83ee142a-b189-4a64-9c7c-e3f693846722 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137732025-04-30T06:42:57.296ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 2351 local_addr = [::1]:33598 method = GET remote_addr = [::1]:45056 req_id = b5f08967-a77d-4503-b9aa-54a3c0735978 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137742025-04-30T06:42:57.296ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 2087 local_addr = [::1]:33598 method = GET remote_addr = [::1]:58190 req_id = c0dc95c9-47c3-41d9-9ed2-a4a4d6826eb5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137752025-04-30T06:42:57.296ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 2002 local_addr = [::1]:33598 method = GET remote_addr = [::1]:48197 req_id = 006ac379-c225-422b-b234-4b78832098e9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137762025-04-30T06:42:57.296ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 1842 local_addr = [::1]:33598 method = GET remote_addr = [::1]:64323 req_id = 6fed0ce1-a9a2-43eb-b7b6-955134411a16 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137772025-04-30T06:42:57.296ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 1788 local_addr = [::1]:33598 method = GET remote_addr = [::1]:56371 req_id = d51da3b4-602d-47f6-a3fd-3630ec69ae36 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137782025-04-30T06:42:57.296ZINFOtest_delete_snapshot (dropshot): request completed kind = pantry latency_us = 1777 local_addr = [::1]:33598 method = GET remote_addr = [::1]:43274 req_id = d79341df-2884-43cd-8261-4985ceae3f10 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
137792025-04-30T06:42:57.301ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137802025-04-30T06:42:57.301ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
137812025-04-30T06:42:57.302ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137822025-04-30T06:42:57.302ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 2 table_name = fields_string
137832025-04-30T06:42:57.302ZTRACtest_delete_snapshot (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
137842025-04-30T06:42:57.314ZDEBGtest_delete_snapshot (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 35191eb6-dc2f-4fbf-b9b0-815a1e904d69 n_rows = 1 table_name = measurements_i64
137852025-04-30T06:42:57.314ZTRACtest_delete_snapshot (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137862025-04-30T06:42:57.314ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a61082b-79b3-44c5-bb1f-32d55ae4e039", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
137872025-04-30T06:42:57.315ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0
137882025-04-30T06:42:57.350ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70ef7c3a-1fa6-45aa-be9f-3733cda277bb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
137892025-04-30T06:42:57.357ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0
137902025-04-30T06:42:57.386ZWARNtest_delete_snapshot (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
137912025-04-30T06:42:57.387ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43f3872e-8b34-4f68-9df0-a88548c918fd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
137922025-04-30T06:42:57.387ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1
137932025-04-30T06:42:57.390ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1e8d7eb-48fc-4503-a046-4a02f8df7344", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
137942025-04-30T06:42:57.390ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0
137952025-04-30T06:42:57.399ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df5e9654-ad42-41a7-8188-34a3701088f1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
137962025-04-30T06:42:57.399ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1
137972025-04-30T06:42:57.403ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
137982025-04-30T06:42:57.406ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45711845-f3dc-43b2-9852-bd967b08602d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
137992025-04-30T06:42:57.407ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0
138002025-04-30T06:42:57.411ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5782fb71-45f0-41d3-ba35-c6faaf500ef2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138012025-04-30T06:42:57.412ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1
138022025-04-30T06:42:57.416ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec8fe121-b3b4-4c15-b184-d3a71cbddeea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138032025-04-30T06:42:57.416ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cmpa
138042025-04-30T06:42:57.418ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34f96d98-baee-4889-a07a-9195b52b8520", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138052025-04-30T06:42:57.418ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
138062025-04-30T06:42:57.419ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b67480f0-1da6-4f25-9775-76d8d04ef340", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138072025-04-30T06:42:57.419ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
138082025-04-30T06:42:57.422ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f46cd358-a443-4d55-b705-942853ed0dba", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138092025-04-30T06:42:57.422ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/0/component/rot/cfpa
138102025-04-30T06:42:57.423ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45c45260-1677-46d6-8ded-20d92ed8e686", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138112025-04-30T06:42:57.423ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1
138122025-04-30T06:42:57.425ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ebca3c2-12ae-4409-b020-f2a630db2de2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138132025-04-30T06:42:57.425ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0
138142025-04-30T06:42:57.427ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93e4416c-c387-4424-92dd-64f432ac72ef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138152025-04-30T06:42:57.427ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1
138162025-04-30T06:42:57.430ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3e55549-b678-4bc2-9a78-5662c782f777", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138172025-04-30T06:42:57.430ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0
138182025-04-30T06:42:57.439ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "837f3153-285b-4937-b799-4ba0e986e88f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138192025-04-30T06:42:57.439ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1
138202025-04-30T06:42:57.450ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f3d9ccb-e9b9-4ad6-910f-d8d1279da996", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138212025-04-30T06:42:57.451ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0
138222025-04-30T06:42:57.453ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9740dcb2-1933-4c50-90ac-1570e95a9374", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138232025-04-30T06:42:57.454ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1
138242025-04-30T06:42:57.455ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138252025-04-30T06:42:57.455ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138262025-04-30T06:42:57.455ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138272025-04-30T06:42:57.455ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
138282025-04-30T06:42:57.456ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138292025-04-30T06:42:57.461ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64de4765-14b8-4c46-909f-902640364d38", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138302025-04-30T06:42:57.461ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cmpa
138312025-04-30T06:42:57.462ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138322025-04-30T06:42:57.472ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138332025-04-30T06:42:57.472ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138342025-04-30T06:42:57.472ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
138352025-04-30T06:42:57.474ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138362025-04-30T06:42:57.479ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84e8b1d7-deaf-4ede-850c-76474e146864", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138372025-04-30T06:42:57.481ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
138382025-04-30T06:42:57.482ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138392025-04-30T06:42:57.482ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
138402025-04-30T06:42:57.484ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138412025-04-30T06:42:57.486ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b78c2fa-d67a-4287-a130-2f3d32d7f454", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138422025-04-30T06:42:57.486ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
138432025-04-30T06:42:57.488ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9981b66-d7f1-421e-a838-b44c92dffbb8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138442025-04-30T06:42:57.488ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/switch/1/component/rot/cfpa
138452025-04-30T06:42:57.492ZINFOtest_delete_snapshot (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
138462025-04-30T06:42:57.493ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138472025-04-30T06:42:57.494ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 685e21a5-9d06-4f4b-9353-2e36e6065c7c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138482025-04-30T06:42:57.494ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 685e21a5-9d06-4f4b-9353-2e36e6065c7c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138492025-04-30T06:42:57.494ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 685e21a5-9d06-4f4b-9353-2e36e6065c7c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138502025-04-30T06:42:57.495ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 685e21a5-9d06-4f4b-9353-2e36e6065c7c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138512025-04-30T06:42:57.497ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0890c17c-3e38-47d2-bb98-036d7b1a05cb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138522025-04-30T06:42:57.497ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0
138532025-04-30T06:42:57.500ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 6241 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 685e21a5-9d06-4f4b-9353-2e36e6065c7c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138542025-04-30T06:42:57.500ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "685e21a5-9d06-4f4b-9353-2e36e6065c7c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138552025-04-30T06:42:57.501ZDEBGtest_delete_snapshot (oximeter): client request body = None method = GET uri = http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
138562025-04-30T06:42:57.501ZTRACtest_delete_snapshot (dropshot_internal): incoming request local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 587ef936-9781-4d99-861b-8effa28c81f0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
138572025-04-30T06:42:57.502ZTRACtest_delete_snapshot (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 587ef936-9781-4d99-861b-8effa28c81f0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
138582025-04-30T06:42:57.502ZDEBGtest_delete_snapshot (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 587ef936-9781-4d99-861b-8effa28c81f0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
138592025-04-30T06:42:57.502ZDEBGtest_delete_snapshot (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 587ef936-9781-4d99-861b-8effa28c81f0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
138602025-04-30T06:42:57.503ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "348ea910-4048-4def-aceb-0d03fb77d649", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138612025-04-30T06:42:57.504ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0
138622025-04-30T06:42:57.506ZINFOtest_delete_snapshot (dropshot_internal): request completed latency_us = 5246 local_addr = 127.0.0.1:35404 method = GET remote_addr = 127.0.0.1:41747 req_id = 587ef936-9781-4d99-861b-8effa28c81f0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D
138632025-04-30T06:42:57.507ZDEBGtest_delete_snapshot (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:35404/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZTNjN2M4My04MjUyLTQzZjQtYmEyZS1jYmQ5MjAwNGM0YjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "587ef936-9781-4d99-861b-8effa28c81f0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138642025-04-30T06:42:57.507ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138652025-04-30T06:42:57.507ZTRACtest_delete_snapshot (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = de3c7c83-8252-43f4-ba2e-cbd92004c4b0
138662025-04-30T06:42:57.507ZINFOtest_delete_snapshot (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
138672025-04-30T06:42:57.509ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf383f77-bb47-49ae-b89c-f9eab6aee82a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138682025-04-30T06:42:57.510ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1
138692025-04-30T06:42:57.513ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f244303-e080-4c91-9f9d-7e0bbaad5fac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138702025-04-30T06:42:57.513ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0
138712025-04-30T06:42:57.516ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0775fab3-cfaa-4845-8723-5c1a1d2c8b30", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138722025-04-30T06:42:57.516ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1
138732025-04-30T06:42:57.523ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
138742025-04-30T06:42:57.527ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2becae9-26e6-40b0-95eb-b5c8358d19c8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138752025-04-30T06:42:57.527ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0
138762025-04-30T06:42:57.530ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f00b04c6-0bae-47c1-84ad-19a6e5b354a4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138772025-04-30T06:42:57.530ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1
138782025-04-30T06:42:57.532ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8576d477-5fd3-4497-87ed-7ecc1831d89d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138792025-04-30T06:42:57.533ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cmpa
138802025-04-30T06:42:57.536ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62b86488-2b96-42cb-bd4d-86430bca017e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138812025-04-30T06:42:57.536ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
138822025-04-30T06:42:57.537ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70e00407-0d5d-4c43-a6fb-d613fdd62fad", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138832025-04-30T06:42:57.537ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
138842025-04-30T06:42:57.538ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40dfea4c-f57a-4a09-9703-9938841501a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138852025-04-30T06:42:57.538ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/0/component/rot/cfpa
138862025-04-30T06:42:57.539ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4465247-0870-4cfb-8b6b-48f54d29a053", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138872025-04-30T06:42:57.539ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1
138882025-04-30T06:42:57.540ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07f83c4a-b6a2-4a17-8373-0cad5d5568c7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138892025-04-30T06:42:57.540ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0
138902025-04-30T06:42:57.542ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f0594fd-9a67-4808-9ab1-2a8925317e42", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138912025-04-30T06:42:57.542ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1
138922025-04-30T06:42:57.544ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5b790d0-c364-42c4-af61-8eb8e6bdcbe3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138932025-04-30T06:42:57.544ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0
138942025-04-30T06:42:57.547ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3536b69e-9a8e-4101-83ba-baca8137809e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138952025-04-30T06:42:57.547ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1
138962025-04-30T06:42:57.550ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6aa5756-7928-40e9-8786-ce2c2254a28a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
138972025-04-30T06:42:57.550ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0
138982025-04-30T06:42:57.552ZDEBGtest_delete_snapshot (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138992025-04-30T06:42:57.552ZDEBGtest_delete_snapshot (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139002025-04-30T06:42:57.554ZTRACtest_delete_snapshot (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139012025-04-30T06:42:57.554ZDEBGtest_delete_snapshot (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139022025-04-30T06:42:57.554ZTRACtest_delete_snapshot (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139032025-04-30T06:42:57.554ZTRACtest_delete_snapshot (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139042025-04-30T06:42:57.555ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94f68c68-6d17-4363-b23f-920a3db6a44f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
139052025-04-30T06:42:57.555ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1
139062025-04-30T06:42:57.558ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9e600f7-f46c-4327-a942-b4abd52dc465", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
139072025-04-30T06:42:57.558ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cmpa
139082025-04-30T06:42:57.562ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7deffc69-4e40-48f4-8251-9426714ce600", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
139092025-04-30T06:42:57.562ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
139102025-04-30T06:42:57.563ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cd84ef5-2869-4ceb-bcb3-ed5e2315f5fe", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
139112025-04-30T06:42:57.564ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
139122025-04-30T06:42:57.565ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb1b6a47-f291-4871-8700-0856740ae6cc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
139132025-04-30T06:42:57.565ZDEBGtest_delete_snapshot (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:35772 method = GET uri = http://[::1]:35772/sp/sled/1/component/rot/cfpa
139142025-04-30T06:42:57.566ZDEBGtest_delete_snapshot (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:35772 result = Ok(Response { url: "http://[::1]:35772/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1998f41b-6a8b-4a0e-a1ff-4031c3a77a40", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
139152025-04-30T06:42:57.566ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
139162025-04-30T06:42:57.566ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139172025-04-30T06:42:57.566ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139182025-04-30T06:42:57.566ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139192025-04-30T06:42:57.567ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139202025-04-30T06:42:57.571ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139212025-04-30T06:42:57.575ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
139222025-04-30T06:42:57.575ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
139232025-04-30T06:42:57.575ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139242025-04-30T06:42:57.575ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139252025-04-30T06:42:57.575ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139262025-04-30T06:42:57.575ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139272025-04-30T06:42:57.576ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
139282025-04-30T06:42:57.576ZTRACtest_delete_snapshot (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139292025-04-30T06:42:57.576ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139302025-04-30T06:42:57.576ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
139312025-04-30T06:42:57.576ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139322025-04-30T06:42:57.578ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139332025-04-30T06:42:57.586ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
139342025-04-30T06:42:57.586ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139352025-04-30T06:42:57.586ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139362025-04-30T06:42:57.588ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139372025-04-30T06:42:57.589ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139382025-04-30T06:42:57.595ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
139392025-04-30T06:42:57.596ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139402025-04-30T06:42:57.596ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
139412025-04-30T06:42:57.596ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139422025-04-30T06:42:57.605ZTRACtest_delete_snapshot (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139432025-04-30T06:42:57.605ZDEBGtest_delete_snapshot (InternalApi): roles roles = RoleSet { roles: {} }
139442025-04-30T06:42:57.606ZDEBGtest_delete_snapshot (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139452025-04-30T06:42:57.690ZDEBGtest_delete_snapshot (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:62607
139462025-04-30T06:42:57.690ZDEBGtest_delete_snapshot (BackgroundTasks): client request SledAgent = http://[::1]:62607 background_task = inventory_collection body = None method = GET uri = http://[::1]:62607/inventory
139472025-04-30T06:42:57.691ZINFOtest_delete_snapshot (dropshot): accepted connection local_addr = [::1]:62607 remote_addr = [::1]:55616 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
139482025-04-30T06:42:57.692ZTRACtest_delete_snapshot (dropshot): incoming request local_addr = [::1]:62607 method = GET remote_addr = [::1]:55616 req_id = d3784a4c-32a2-4601-b368-5d0b5271c49b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
139492025-04-30T06:42:57.693ZINFOtest_delete_snapshot (dropshot): request completed latency_us = 700 local_addr = [::1]:62607 method = GET remote_addr = [::1]:55616 req_id = d3784a4c-32a2-4601-b368-5d0b5271c49b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
139502025-04-30T06:42:57.693ZDEBGtest_delete_snapshot (BackgroundTasks): client response SledAgent = http://[::1]:62607 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:62607/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3784a4c-32a2-4601-b368-5d0b5271c49b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:42:56 GMT"} })
139512025-04-30T06:42:57.696ZDEBGtest_delete_snapshot (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
139522025-04-30T06:42:57.696ZDEBGtest_delete_snapshot (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
139532025-04-30T06:42:57.696ZDEBGtest_delete_snapshot (BackgroundTasks): finished collection background_task = inventory_collection
139542025-04-30T06:42:57.698ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
139552025-04-30T06:42:57.698ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139562025-04-30T06:42:57.698ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139572025-04-30T06:42:57.698ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139582025-04-30T06:42:57.699ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139592025-04-30T06:42:57.733ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139602025-04-30T06:42:57.765ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
139612025-04-30T06:42:57.768ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139622025-04-30T06:42:57.768ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139632025-04-30T06:42:57.768ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139642025-04-30T06:42:57.784ZDEBGtest_delete_snapshot (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
139652025-04-30T06:42:57.784ZWARNtest_delete_snapshot (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
139662025-04-30T06:42:57.787ZWARNtest_delete_snapshot (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
139672025-04-30T06:42:57.832ZDEBGtest_delete_snapshot: attempt to fetch latest inventory collection result = Ok("found")
139682025-04-30T06:42:57.851ZWARNtest_delete_snapshot (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
139692025-04-30T06:42:57.852ZWARNtest_delete_snapshot (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
139702025-04-30T06:42:57.852ZDEBGtest_delete_snapshot (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.168526863s iteration = 50 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
139712025-04-30T06:42:57.852ZDEBGtest_delete_snapshot (BackgroundTasks): activating background_task = inventory_collection iteration = 51 reason = Signaled
139722025-04-30T06:42:57.852ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
139732025-04-30T06:42:57.852ZTRACtest_delete_snapshot (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139742025-04-30T06:42:57.852ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139752025-04-30T06:42:57.852ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139762025-04-30T06:42:57.853ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139772025-04-30T06:42:57.857ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139782025-04-30T06:42:57.862ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
139792025-04-30T06:42:57.863ZTRACtest_delete_snapshot (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139802025-04-30T06:42:57.863ZDEBGtest_delete_snapshot (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139812025-04-30T06:42:57.863ZDEBGtest_delete_snapshot (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())